1. Using Oracle Infrastructure Monitoring
  2. Troubleshooting
  3. Lack of Data

Lack of Data

Issues encountered when using Infrastructure Monitoring typically fall into the following categories of lack of data in performance charts and metric collection errors.

Lack of Data in Performance Charts

For Cloud agent-monitored entities:

  1. Check the time period in the UI to verify it includes the time period where data is expected.

  2. Check if the entity it up.   If the entity is down,  no metrics can be collected.

  3. Find out if the agent is up.

    1. Go to the agent homepage.

    2. Determine the current status of the agent via the agent command line utility.   omcli status agent  

      $ ./omcli status agent
Oracle Management Cloud Agent 
Copyright (c) 1996, 2017 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Version : 1.27.0
State Home : /scratch2/agent/den00yla/lama_agent/agent_inst
Log Directory : /scratch2/agent/den00yla/lama_agent/agent_inst/sysman/log
Binaries Location : /scratch2/agent/den00yla/lama_agent/core/1.27.0
Process ID : 85600
Parent Process ID : 90122
URL : https://myhost.myco.com:58858/emd/main/
Started at : 2018-02-21 08:26:28
Started by user : mahessub
Operating System : Linux version 3.8.13-118.20.2.el6uek.x86_64 (amd64)
Data Collector enabled : false
Sender Status : FUNCTIONAL
Gateway Upload Status : FUNCTIONAL
Last successful upload : 2018-02-23 16:04:03
Last attempted upload : 2018-02-23 16:04:03
Pending Files (MB) : 0
Pending Files : 1
Backoff Expiration : (none)

---------------------------------------------------------------
Agent is Running and Ready
      Ensure the following:

      • The agent status message displays Agent is Running and Ready.

      • Last successful upload shows a recent timestamp.
      • Pending Files (MB) and Pending Files do not show large values. If either of these statistics is large, the data may have not been uploaded yet. You should run the omcli status agent command periodically to verify that Pending Files (MB) and Pending Files are decreasing.

    3. Check whether the agent is able to collect the metric(s) whose data you are not seeing in the performance charts.

      1. Run the omcli getmetric agent command: 

        omcli getmetric agent TARGETNAME,TARGETTYPE,METRICNAME

        Note:

        Make sure there are no spaces before or after comma delimiters.

        If the command returns values, then it means the agent is able to collect the data.

        $ ./omcli getmetric agent myhost.myco.com,omc_host_linux,HOST_CPU

Oracle Management Cloud Agent 
Copyright (c) 1996, 2017 Oracle Corporation. All rights reserved.
idleTimeRaw,userTimeRaw,systemTimeRaw,waitTimeRaw,hardInterruptsTimeRaw,softInterruptsTimeRaw,cpuStealTimeRaw,idleTimeRawDiff,userTimeRawDiff,systemTimeRawDiff,waitTimeRawDiff,hardInterruptsTimeRawDiff,softInterruptsTimeRawDiff,cpuStealTimeRawDiff,totalTimeRawDiff,cpuUserModePercent,cpuSystemModePercent,cpuUsageSec,cpuUtilizationPercent,cpuStolenPercent,cpuIdlePercent,cpuLoad1min,cpuLoad5min,cpuLoad15min,intervalSec
701388434,5781055,2644520,563349,135,12397,84674,10114,349,50,3,0,0,1,10517,3.318,0.475,399,3.794,0.01,96.168,0.005,0.015,0.0125,26

      2. Use the Agent Metric Browser to view the data the Cloud agent collects from various entities. See .

  4. Check if there are metric collection error alerts on the entity home page. If there are problems collecting the metric, then no data will appear. You should see a metric collection error (Alert Type = Metric Error). To fix the metric error, ensure that all of the prerequisites for monitoring the entity type have been met.

Metric Collection Errors

Table 10-1 Debugging Metric Collection Errors

Entity Type Metric Collection Error Message Cause/Resolution

Oracle Database

oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Failed to connect: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

DB monitoring relies on a JDBC connection to the database. Check if the DB service is registered with the listener.

Oracle Pluggable Database

oracle.sysman.emd.fetchlets.db.exception.AwrFetchletException: SharedDBFetchlet - E8C3A7030E443414D10A7BAF993842BD - AWR Shared metric collection failed: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

DB monitoring relies on a JDBC connection to the database. Ensure that the DB service is registered with the listener.

Oracle Database

 oracle.sysman.emd.fetchlets.db.exception.AwrFetchletException: UDSSqlEntityFetchlet - E8C3A7030E443414D10A7BAF993842BD - AWR UDS sql entity collection failed: java.sql.SQLRecoverableException: IO Error: Socket read timed out

The root cause in most cases is a failure of the database to respond in a reasonable amount of time. This error could also be caused by network issues.

Oracle Database

orcldb123/PDB1(Oracle Pluggable Database) is in Error state. Reason :  oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: assoc target  does not exist or is broken

apmc/ORCLPDB(Oracle Pluggable Database) is in Error state. Reason : oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: assoc target does not exist or is broken

This issue occurs when the wrong JSON file has been used.

Example: The CDB JSON was used to discover a PDB.

For more information about Infrastructure Monitoring JSON files, see Download and Customize Oracle Infrastructure Monitoring JSONs.

Oracle Database

Error evaluating Top SQL Executions:5MinCollection - AwrFetchletException: SharedDBFetchlet - 4A8DD99203A4341B15B61BA452994851 - internal error: java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

 Ensure the monitoring user has sufficient privileges. For more information, see Prerequisites and Monitoring Credentials.

 Oracle Database

Error evaluating CorrelationAssociations:24HrCollection-FetchletException: ORA-24247:network access denied by access control list (ACL) ORA-06512:at “SYS.UTL_INADD”,line 19ORA-06512:at”SYS.UTL_INADDR",line 40 ORA-06512:at line 1

Run the PL/SQL block

dbms_network_acl_admin

For more information about Oracle Database prerequisites, see Prerequisites and Monitoring Credentials.

Oracle Database Error evaluating CorrelationAssociations:24HrCollection-FetchletException: ORA24247: network access denied by access control list (ACL) ORA-06512:at”SYS.UTL_INADDR”,line 19 ORA-06512: at “SYS.UTL_INADDR”, Line 40 ORA-06512: at line 1

"Error evaluating SQL statistics..."

The Oracle DB target needs to have DB Diagnostic Pack enabled or be an Oracle DB Enterprise Edition.

Tomcat

tomcat_mzlee (Tomcat) is in an Error state.

tomcat_rest (Tomcat) is in an Error state.

  

Host (Linux)

Criticaloracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Result has repeating key value : lo

 

No Metric Data Appears in the UI

Steps 1 and 2 only need to be performed if the entity was newly added. If the entity was added in the past, but suddenly is not showing any metric data, complete steps 3 through 8.

  1. Make sure you have completed all the prerequisite steps. See Prerequisites and Monitoring Credentials.

  2. Check to see if the entity type and version is supported. See Supported Entity Types.

  3. Go to the entity’s home page and see if there are any metric collection errors.

  4. Check that the entity is up.

  5. Check that the agent is up.

  6. Run the metrics directly using the getmetric command.

    omcli getmetric agent TARGETNAME,TARGETTYPE,METRICNAME

    To get the targetname, targettype:

    config agent listtargets (confirm that you get the target name and type)

    OPEN: metricname (this corresponds to the metricGroupName)

  7. Check metrics in the agent metric browser.

  8. Contact Oracle Support and provide debug information using the following command:

    omcli generate_support_bundle agent DIRECTORY

Metrics Not Collected for Windows Environments

A Cloud agent is deployed on a Windows Server. Although the host entity has been registered in Infrastructure Monitoring performance metrics are not collected.  In this situation, the root cause of the issue is performance counters in this environment were corrupt. . 

To check whether performance counters are functioning properly:

  1. From an administrator command prompt execute wbemtest. A dialog displays.

  2. Click connect. Check the namespace is root\cimv2 and click connect again.

  3. Click Open class and enter the class name. For example: Win32_PerfRawData_PerfOS_Processor

  4. A new window will open with details about the class. If the window displays the class details, then the performance counters are working properly.

  5. If any error message is displayed, execute following command as administrator from command prompt:
    lodctr /R

  6. After executing the above command, perform steps one through four again to verify the performance counters are working.