28.3 Locating Management Agent Log and Trace Files

The following sections provide information on the log and trace files for the Oracle Management Agent:

28.3.1 About the Management Agent Log and Trace Files

Oracle Management Agent log and trace files store important information that support personnel can later use to troubleshoot problems. The agent main log is located in $EMSTATE/sysman/log. The log is segmented by default to 11 segments, 5MB each. The segments are named gcagent.log and gcagent.log.# where # is a number in the range of 1-10. These settings are controlled by properties in emd.properties as explained in the following sections. The latest segment is always gcagent.log and the oldest is the gcagent.log.X where X is the highest number.

The Management Agent uses the following log files:

  • Oracle Management Agent metadata log file (gcagent.log)

    This log file contains trace, debug, information, error, or warning messages from the agent.

  • Oracle Management Agent fetchlet trace file (gcagent_sdk.trc)

    This log file contains logging information about fetchlets and receivelets.

  • Oracle Management Agent errors log file (gcagent_errors.log)

    This error log file contains information about errors. The errors in this file are duplicate of the errors in gcagent.log.

  • Oracle Management Agent metadata log file (gcagent_mdu.log)

    This log tracks the metadata updates to the agent.

  • Enterprise Manager Control log file (emctl.log)

    The information is saved to emctl.log file, when you run the Enterprise Manager Control commands. For more information about emctl.log file, see chapter Starting and Stopping Enterprise Manager Components.

Note:

All the agent logs mentioned above (existing in $EMSTATE/sysman/log) are transient. Agent logs are segmented and have a limited overall size and hence need not be deleted or managed.

28.3.1.1 Structure of Agent Log Files

The log contain individual log messages with the following format:

YYYY-MM-DD HH:MM:SS,### [<tid>:<thread code or code:name>] <level> -<the message>

Where:

  • YYYY-MM-DD HH:MM:SS,### is a timestamp (in 24 hours format and ### is the fraction in msec).

  • <tid> is the thread id (as a decimal number)

  • <thread name or code> is the thread full name or an abbreviated hexadecimal code (see the following example).

  • <level> is the logging level that can be one of (in ascending order of importance): DEBUG, INFO, WARN, ERROR, FATAL.

  • <the message> is the free text message that is being logged. The message can contain new lines and spawn multiple lines.

For example:

2011-06-07 15:00:00,016 [1:3305B9:main] DEBUG - ADR_BASE='/ade/example_user/oracle/example/agentStateDir' 
2011-06-07 15:00:01,883 [1:3305B9] INFO - Agent is starting up

28.3.2 Locating the Management Agent Log and Trace Files

The log and trace files for the Management Agent are written in the agent runtime directory. You can find the runtime directory by using this command:

$ emctl getemhome

The log and trace files will be located at <EMHOME>/sysman/log.

28.3.3 Setting Oracle Management Agent Log Levels

Every log message is logged using a specific log level. The log levels are ordered in priority order: DEBUG, INFO, WARN, ERROR, and FATAL. The log setting determines the minimum level that will be included in the log. For example, if the log level is set to INFO (the default), only log messages of level INFO and above (INFO, WARN, ERROR and FATAL) are going to be included in the log.

The logging configuration syntax uses the concept of handlers (appendares in log4j terms) and loggers. A handler defines a single output file and how the file is to be managed (maximum file size, number of segments, and so on). Note that there is a default logging prefix oracle.sysman that is used for all handlers that does not specify any logging prefix. The logging properties uses the Logger. prefix for agent (log4j) logging configuration and ODLLogger. prefix for the ODL (which is based on java.util.logger.*) logging configuration. Beside the prefix, both systems share the same syntax. The configuration full syntax (without a Logger or ODLLogger prefix) is the following:

Table 28-2

Property Name Description Mandatory Default Value

directory=<directory>

Defines the logging system (log4j or ODL) logging directory. Specifing a directoryfor one system does not affect the other system (setting Logger. directory will only affect the Logger. configuration but not ODLLogger.)

No

$EMSTATE/sysman/log

<handler>.filename=<filename>

The filename to use for the handler. If the filename is relative it will be relative to the logging directory (see direrctory property above). An absolute file name will be used as is.

Yes

 

<handler>.level=<level>

The default logging level for the handler. Possbile levels are:

DEBUG, INFO, WARN, ERROR, FATAL

Yes

 

<handler>.totalSize=<size>

The total size in MB for all the handler file segments.

No

No limit

<handler>.segment.count=<count>

The number of segments to use for the handler.

No

1

<handler>.logger=<logger names>

A comma delimited list of logger names that will use this handler.

No

When not specified, the default logger is used.

level.<logger name>=<level>

Set a specific logging level to the logger and all its descendants. Possbile levels are:

DEBUG, INFO, WARN, ERROR, FATAL

No

 

additivity.<logger name>=<true or false>

If set to false, only handlers that are configured for the specific logger name will be used. Otherwise, handlers that are configured for the logger parent name will also be used.

No

true

An example of the syntax is as follows:

# logging properties
Logger.log.filename=gcagent.log
Logger.log.level=INFO
Logger.log.totalSize=100
Logger.log.segment.count=20 

ODLLogger.wsm.level=ERROR
ODLLogger.wsm.totalSize=5
ODLLogger.wsm.segment.count=5
ODLLogger.wsm.filename=gcagent_wsm.log

The above log configuration sets up a handler (log) that creates a gcagent.log file (in the default logging directory) with a default logging level of INFO, total size of 100MB, uses up to 20 segments, and is configured to be used by the default logger (oracle.sysman).

28.3.3.1 Modifying the Default Logging Level

To enable DEBUG level logging for the Management Agent, set the log handler level to DEBUG (see below). And then reload the agent.

Logger.log.level=DEBUG 

Alternatively, use emctl setproperty agent command as follows:

$ emctl setproperty agent -name "Logger.log.level" -value DEBUG

or

$ emctl setproperty agent -name "Logger.log.level" -value "DEBUG"

28.3.3.2 Setting gcagent.log

The gcagent.log is the agent main log that contain log entries from all the agent core code. The following is gcagent.log configuration:

Logger.log.filename=gcagent.log
Logger.log.level=DEBUG
Logger.log.totalSize=100
Logger.log.segment.count=20

28.3.3.3 Setting gcagent_error.log

The gcagent_errors.log is a subset of the gcagent.log and contains log messages of ERROR and FATAL levels. The logging configuration for gcagent_errors.log is specified in emd.properties. Following are the settings for gcagent_errors.log:

Logger.err.filename=gcagent_errors.log
Logger.err.level=ERROR
Logger.err.totalSize=100
Logger.err.segment.count=5

28.3.3.4 Setting the Log Level for Individual Classes and Packages

The logging level for individual class and/or packages can also be set. The following are examples that are currently configured by default:

# Set the class loaders to level INFO
Logger.level.oracle.sysman.gcagent.metadata.impl.ChainedClassLoader=INFO 
Logger.level.oracle.sysman.gcagent.metadata.impl.ReverseDelegationClassLoader=INFO 
Logger.level.oracle.sysman.gcagent.metadata.impl.PluginLibraryClassLoader=INFO 
Logger.level.oracle.sysman.gcagent.metadata.impl.PluginClassLoader=INFO 

The above configuration changed the default level of logging for the four classes to be INFO. When the default level of logging is INFO it does not make any difference but if the default log level is set to DEBUG (when debugging the code) it will prevent those four classes from logging at DEBUG level (as they are normally too verbose).

The reverse is also true, for example if the following configuration is added (not set by default):

Logger.level.oracle.sysman.gcagent.metadata.impl.collection=DEBUG 

It will cause all classes in the oracle.sysman.gcagent.metadata.impl.collection package to log at DEBUG level even if the default log level is INFO.

28.3.3.5 Setting gcagent_mdu.log

A set of entries are created in the gcagent_mdu.log file for each client command that modifies target instances, target instance collections, or blackouts. Entries are as follows:

2011-08-18 22:56:40,467 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SAVE TARGET(S)
<Target IDENTIFIER="TARGET_GUID=6A3A159D0BB320C50B7926E0671A1A98" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="EM Management Beacon" NAME="EM Management Beacon" TYPE="oracle_beacon"/>
<Target IDENTIFIER="TARGET_GUID=51F9BBC6F5B833058F4278B51E496000" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="mytestBeacon" NAME="mytestBeacon" TYPE="oracle_beacon"><Property VALUE="***" NAME="proxyHost"/><Property VALUE="***" NAME="proxyPort"/><Property VALUE="***" NAME="dontProxyFor"/></Target>
<Target IDENTIFIER="TARGET_GUID=7C4336B536C9F241DBCAC4D1D082AD22" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="CSAcollector" NAME="CSAcollector" TYPE="oracle_csa_collector"><Property VALUE="***" NAME="recvFileDir"/></Target>
<Target IDENTIFIER="TARGET_GUID=207B57A3FE300C86F81FE7D409F5DD1C" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="Oemrep_Database" NAME="Oemrep_Database" TYPE="oracle_database"><Property VALUE="***" NAME="MachineName"/><Property VALUE="***" NAME="Port"/><Property VALUE="***" NAME="SID"/><Property VALUE="***" NAME="OracleHome"/><Property ENCRYPTED="FALSE" VALUE="***" NAME="UserName"/><Property ENCRYPTED="FALSE" VALUE="***" NAME="Role"/><Property ENCRYPTED="FALSE" VALUE="***" NAME="password"/></Target>
<Target IDENTIFIER="TARGET_GUID=0C48C5AE0FAFB42ED91F897FF398FC84" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="Management Services and Repository" NAME="Management Services and Repository" TYPE="oracle_emrep"><Property VALUE="***" NAME="ConnectDescriptor"/><Property ENCRYPTED="FALSE" VALUE="***" NAME="UserName"/><Property ENCRYPTED="FALSE" VALUE="***" NAME="password"/><AssocTargetInstance ASSOC_TARGET_TYPE="oracle_oms" ASSOC_TARGET_NAME="linuxserver07.myco.com:41034_Management_Service" ASSOCIATION_NAME="app_composite_contains"/><AssocTargetInstance ASSOC_TARGET_TYPE="oracle_oms" ASSOC_TARGET_NAME="linuxserver07.myco.com:41034_Management_Service" ASSOCIATION_NAME="internal_contains"/><CompositeMembership><Member ASSOCIATION="" NAME="linuxserver07.myco.com:41034_Management_Service_CONSOLE" TYPE="oracle_oms_console"/><Member ASSOCIATION="" NAME="linuxserver07.myco.com:41034_Management_Service_PBS" TYPE="oracle_oms_pbs"/><Member ASSOCIATION="" NAME="linuxserver07.myco.com:41034_Management_Service" TYPE="oracle_oms"/></CompositeMembership></Target>
<Target IDENTIFIER="TARGET_GUID=DF64B4A7C0F2EEBA7894EA3AD4CAF61E" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="linuxserver07.myco.com:41034_Management_Service" NAME="linuxserver07.myco.com:41034_Management_Service" TYPE="oracle_oms"><Property VALUE="***" NAME="InstanceHome"/><Property VALUE="***" NAME="OracleHome"/><AssocTargetInstance ASSOC_TARGET_TYPE="oracle_oms_console" ASSOC_TARGET_NAME="linuxserver07.myco.com:41034_Management_Service_CONSOLE" ASSOCIATION_NAME="app_composite_contains"/><AssocTargetInstance ASSOC_TARGET_TYPE="oracle_oms_pbs" ASSOC_TARGET_NAME="linuxserver07.myco.com:41034_Management_Service_PBS" ASSOCIATION_NAME="app_composite_contains"/><AssocTargetInstance ASSOC_TARGET_TYPE="oracle_oms_console" ASSOC_TARGET_NAME="linuxserver07.myco.com:41034_Management_Service_CONSOLE" ASSOCIATION_NAME="internal_contains"/><AssocTargetInstance ASSOC_TARGET_TYPE="oracle_oms_pbs" ASSOC_TARGET_NAME="linuxserver07.myco.com:41034_Management_Service_PBS" ASSOCIATION_NAME="internal_contains"/><CompositeMembership><MemberOf ASSOCIATION="" NAME="Management Services and Repository" TYPE="oracle_emrep"/><Member ASSOCIATION="" NAME="linuxserver07.myco.com:41034_Management_Service_CONSOLE" TYPE="oracle_oms_console"/><Member ASSOCIATION="" NAME="linuxserver07.myco.com:41034_Management_Service_PBS" TYPE="oracle_oms_pbs"/></CompositeMembership></Target>
<Target IDENTIFIER="TARGET_GUID=4D290260F13596502EFD8F3E22752404" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="linuxserver07.myco.com:41034_Management_Service_CONSOLE" NAME="linuxserver07.myco.com:41034_Management_Service_CONSOLE" TYPE="oracle_oms_console"><Property VALUE="***" NAME="InstanceHome"/><Property VALUE="***" NAME="OracleHome"/><CompositeMembership><MemberOf ASSOCIATION="" NAME="Management Services and Repository" TYPE="oracle_emrep"/><MemberOf ASSOCIATION="" NAME="linuxserver07.myco.com:41034_Management_Service" TYPE="oracle_oms"/></CompositeMembership></Target>
<Target IDENTIFIER="TARGET_GUID=D0A23AE06A9E678221B075A216364541" STATUS="MONITORED" TIMEZONE_REGION="" ON_HOST="" DISPLAY_NAME="linuxserver07.myco.com:41034_Management_Service_PBS" NAME="linuxserver07.myco.com:41034_Management_Service_PBS" TYPE="oracle_oms_pbs"><Property VALUE="***" NAME="InstanceHome"/><Property VALUE="***" NAME="OracleHome"/><CompositeMembership><MemberOf ASSOCIATION="" NAME="Management Services and Repository" TYPE="oracle_emrep"/><MemberOf ASSOCIATION="" NAME="linuxserver07.myco.com:41034_Management_Service" TYPE="oracle_oms"/></CompositeMembership></Target>
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS
2011-08-18 22:57:10,084 [HTTP Listener-34 - /emd/main/ (DispatchRequests)] - SUCCESS

For the batch of saved targets in the above example, the original request came in at 22:56:40 and the list of targets saved are found in the line(s) following the SAVE TARGET(S) message. In this case, there were 8 targets. The result of saving the targets is available in the next 8 lines (for the same thread) and in this case all were saved successfully by 22:57:10.

The pattern is the same for saved collection items (or collections) and blackouts.

The logging configuration for the gcagent_mdu log is specified in emd.properties but you must not modify this log. For example, these entries are logged at INFO level, which means that if you decided to save space and change this to WARN only by editing the mdu log entries in the emd.properties file, you will have effectively disabled this log.

Following are the settings for gcagent_mdu log:

Logger.mdu.filename=gcagent_mdu.log
Logger.mdu.level=INFO
Logger.mdu.totalSize=100
Logger.mdu.segment.count=5
Logger.mdu.logger=Mdu

Note:

Change the filename and logger settings only if asked by Support.

28.3.3.6 Setting the TRACE Level

The following _enableTrace property when set to "true" will enable the TRACE logging level that shows as DEBUG messages.

Logger._enableTrace=true

The default log level for the agent log must be set to DEBUG for the tracing level to work.