E Troubleshooting Oracle Application Server Containers for J2EE

This appendix describes tools and methods that can be used to troubleshoot Oracle Application Server Containers for J2EE or any scheduler-based applications. The following topics are covered:

Oracle Diagnostic Logging (ODL)
DMS Metrics
Frequently Asked Questions About Job Scheduler Monitoring
Frequently Asked Questions About Job Scheduler Logging

E.1 Oracle Diagnostic Logging (ODL)

To simplify integration with Oracle Application Server, the standard JDK1.4.1 java.util.logging APIs are used. These APIs make a clear separation of the logging APIs (java.util.logging.Logger) from the APIs that control writing logged messages to various destinations (java.util.logging.Handler) and also from APIs that control message formatting and localization (java.util.logging.Formatter). The message are logged in ODL format using Oracle's ODL handler.

E.1.1 Types of Logging

Job Scheduler provides the following types of logging:

Run time logging, which is performed on behalf of the scheduler subsystem (for example, a warning message as a result of misconfiguration).
Job logging, which is related to a job's execution. There are two types of job logging:
- Implicit Job Logging. This type of logging is primarily performed by Job Scheduler and is specified as part of the job's definition.
- Explicit Job Logging. This type of logging is performed by the actual job implementation, meaning that it is user-defined.

E.1.1.1 Implicit Job Logging

Implicit job logging is specified as part of the job's definition. Because Job Scheduler uses the Java logging APIs, log levels are specified using the log levels provided by the java.util.logging.Level class.

The default level of implicit logging is set at Level.FINER. Job Scheduler uses only a subset of these levels to log messages.

If the logging level is set to Level.WARNING, log entries are written under the following conditions:

Running the job resulted in a JobExecutionException exception.
Running the job resulted in a RuntimeException exception.

If the logging level is set to a value of Level.FINE, the following additional information is written to the log:

Date and time at which the job started.
Date and time at which the job ended.

If the logging level is set to a value of Level.FINER, the following additional information is written to the log:

Date and time at which the associated trigger evaluated a notification, and the result of the evaluation.

If the logging level is set to a value of Level.FINEST, the following additional information is written to the log:

Total elapsed time of the job.

Each log entry contains the following:

Job description
Job implementation class name
Date and time
Stack trace (if the job results in an exception)
Associated message parameters

E.1.1.2 Explicit Job Logging

The same logging facilities used by implicit job logging are also available to the job implementation when the job runs. The logging context is available through the context that is passed to the job when it is run through the JobContext object:

public interface JobContext extends Serializable {   public Job getJob();   public java.util.logging.Logger getLogger();   public java.util.logging.Logger getLogger(String resourceBundleName);}

Either of the getLogger() methods can be used, but the latter method allows a resource bundle to be specified.

E.1.2 Configuring the Global Log Levels

You can configure the global log level of Job Scheduler. For more information, see Section 9.4.

E.1.3 Logging Example

Example E-1 shows how to add logging capabilities to a job. Specifically, an information log entry is written before every file is copied. This is done by retrieving the logger from the job context and writing an informational log message before performing the copy command.

Example E-1 Job Implementation with Logging

import java.io.File;
import java.io.IOException;
import java.util.logging.Logger;
import oracle.ias.scheduler.Job;
import oracle.ias.scheduler.Executable;
import oracle.ias.scheduler.Cancellable;
import oracle.ias.scheduler.JobContext;
import oracle.ias.scheduler.JobCancellationException;
import oracle.ias.scheduler.JobExecutionException;
 
 
public class CancellableBackupJobLogged implements Executable, Cancellable {
 
    boolean m_cancelled = false;
 
    public void cancel() {
        m_cancelled = true;
    }
 
    public void execute(JobContext context) throws JobExecutionException, JobCancellationException {
 
        // retrieve the source/destination directories
        Job job = context.getJob();
        Logger log = context.getLogger();
        String source = job.getProperties().getProperty("SourceDirectory");
        String destination = job.getProperties().getProperty("DestinationDirectory");
 
        // get the list of files to copy
        File directory = new File(source);
        File[] files = directory.listFiles();
        
        // copy the files
        Runtime runtime = Runtime.getRuntime();
        Process process;
        for (int x = 0; x < files.length; x++) {
 
            // cancelled?
            if (m_cancelled) {
                throw new JobCancellationException();
            }
 
            log.info("copying file "+files[x]);
            try {
                process = runtime.exec("/bin/cp " + files[x].toString() +
                                       " " + destination);
                process.waitFor();
            } catch(IOException e) {
                throw new RuntimeException("copy failed: "+files[x],e);
            } catch(InterruptedException e) {
                throw new RuntimeException("copy failed: "+files[x],e);
            }
        }
    }
}

The log level can be set by invoking the appropriate API on the logger. For example:

<code>
      Logger logger = jobContext.getLogger();
      logger.setLoglevel(Level.FINEST);
</code>

Note:

Since all jobs share a single logger instance, setting the log level will affect the logging of all subsequent messages for all instances of the job.

E.2 DMS Metrics

Oracle Dynamic Monitoring Service (DMS) is used to measure application specific performance information. Two types of metrics are provided:

Scheduler metrics. Provides statistics pertaining to a specific Job Scheduler instance (for example, total number of executing job instances, total number of active jobs, or total number of paused jobs).
Job metrics. Provides aggregate job statistics as well as information pertaining to a specific job (for example, the job description, state of the job, or number of failed executions).

Table E-1 lists the information provided by scheduler metrics:

Table E-1 Statistic Types for Scheduler Metrics

Metric Name	Description
`schedulerStartTime`	`System.currentTimeMillis()` when Job Scheduler starts.
`executingJobs`	Total number of job instances that are currently running.
`activeJobs`	Total number of active jobs.
`pausedJobs`	Total number of paused jobs.
`completedJobs`	Total number of completed jobs.

Table E-2 lists the information provided by job metrics:

Table E-2 Statistic Types for JobStats

Metric Name	Description
`jobSchedule`	String version of the schedule.
`jobTrigger`	String version of the trigger.
`jobLogLevel`	Log level.
`jobClassName`	String version of the class name.
`jobDescription`	Job description.
`jobExecutionThreshold`	Execution threshold (in millliseconds).
`jobRetryPeriod`	Retry period (in milliseconds).
`jobState`	State of the job (active, paused, or complete)
`execution`	Duration for which an instance of this job runs.
`failedExecutions`	Number of failed runs.
`cancelledExecutions`	Number of canceled runs since JVM startup.
`successfulExecutions`	Number of successful runs since JVM startup.
`blackoutExecutions`	Number of runs that were blacked out since JVM startup.
`exceededThresholdExecutions`	Number of executions that exceeded the execution threshold since JVM startup.

For more information about DMS, please refer to the Oracle Application Server Performance Guide.

E.3 Frequently Asked Questions About Job Scheduler Monitoring

How do I monitor Job Scheduler activities?

You can connect to the DMS Spy servlet to look at statistics for Job Scheduler and its various jobs (for example, the number of currently active, completed, and currently running jobs). For each job, information is provided about its current state, duration, and result (for example, whether or not the job failed). For detailed information, see the DMS Addendum.

Can I disable DMS statistics collection?

Yes. For more information, see Chapter 9.

E.4 Frequently Asked Questions About Job Scheduler Logging

How do I configure logging for Job Scheduler?

For more information about configuring logging for Job Scheduler, see Chapter 9.

How can I use logging to troubleshoot problems?

To troubleshoot a particular job, you can increase the granularity of the log messages by changing the log level of the particular job in question. This can be accomplished dynamically through the job MBean management interface in the Administration Console, or by directly invoking the remote scheduler interface. The global root logger's default level is set to Level.WARNING.