Logging configuration

Data Processing has a default configuration file, logging.properties, that sets the logging properties.

By default, the logging.properties file is located in the /user/bdd/edp/lib directory. The hdfsEdpLibPath property in the data_processing_CLI file controls the location of this file in HDFS.

The file has the following properties:
Logging property Description
handlers A comma-delimited list of handler class names that are added to the root Logger. The default handlers are java.util.logging.FileHandler and java.util.logging.ConsoleHandler (with a default level of INFO).
java.util.logging.FileHandler.level Sets the log level for all FileHandler instances. The default log level is FINE.
java.util.logging.FileHandler.pattern The log file name pattern. The default is %t/edpLog%u%g.log which means that the file is named edpLog%u%g.log where:
  • %u is a unique number to resolve conflicts between simultaneous Java processes.
  • %g is the generation number to distinguish between rotating logs.
%t specifies the system temporary directory as the location in which the log files are stored.
java.util.logging.FileHandler.limit The maximum size of the file, in bytes. If this is 0, there is no limit. The default is 1000000 (which is 1 MB). Logs larger than 1MB roll over to the next log file.
java.util.logging.FileHandler.count The number of log files to use in the log file rotation. The default is 10000 (which produces a maximum of 10,000 log files).
java.util.logging.FileHandler.formatter The class name of the Formatter to use for the FileHandler instances.
java.util.logging.ConsoleHandler.level Sets the default log level for all ConsoleHandler instances.
java.util.logging.FileHandler.append Specifies whether the FileHandler should append onto any existing files (defaults to false).
java.util.logging.ConsoleHandler.formatter The class name of the Formatter to use for the ConsoleHandler instances.
java.util.logging.SimpleFormatter.format Specifies the format to use for log messages. For details on the format syntax, see: http://docs.oracle.com/javase/7/docs/api/java/util/logging/SimpleFormatter.html

com.oracle.eid = FINE

com.oracle.endeca = FINE

com.oracle.endeca.pdi = INFO

Sets the default logging level for the Big Data Discovery loggers.

org.eclipse.jetty = WARNING org.apache.spark.repl.

SparkIMain$exprTyper = INFO org.apache.spark.repl.SparkILoop

$SparkILoopInterpreter = INFO

Sets the default logging level for the Spark and Jetty loggers.

For details on the FileHandler settings, see http://docs.oracle.com/javase/7/docs/api/java/util/logging/FileHandler.html

Logging levels

The logging level specifies the amount of information that is logged. The levels (in descending order) are:
  • SEVERE — Indicates a serious failure. In general, SEVERE messages describe events that are of considerable importance and which will prevent normal program execution.
  • WARNING — Indicates a potential problem. In general, WARNING messages describe events that will be of interest to end users or system managers, or which indicate potential problems.
  • INFO — A message level for informational messages. The INFO level should only be used for reasonably significant messages that will make sense to end users and system administrators.
  • CONFIG — A message level for static configuration messages. CONFIG messages are intended to provide a variety of static configuration information, and to assist in debugging problems that may be associated with particular configurations.
  • FINE — A message level providing tracing information. All options, FINE, FINER, and FINEST, are intended for relatively detailed tracing. Of these levels, FINE should be used for the lowest volume (and most important) tracing messages.
  • FINER — Indicates a fairly detailed tracing message.
  • FINEST — Indicates a highly detailed tracing message. FINEST should be used for the most voluminous detailed output.
  • ALL — Enables logging of all messages.

These levels allow you to monitor events of interest at the appropriate granularity without being overwhelmed by messages that are not relevant. When you are initially setting up your application in a development environment, you might want to use the FINEST level to get all messages, and change to a less verbose level in production.