NUMERICPRECISION

This Essbase configuration setting specifies the number of precision digits used by Report Writer for numerical comparison.

Syntax

NUMERICPRECISION n

n—Specifies the number of precision digits to be considered in the numerical comparison. Acceptable values for n are -1 through 15. A value of -1 indicates a full comparison. The default value is 4.

Description

This setting defines the number of precision digits used by Report Writer for numerical comparison.

The numeric comparison function subtracts one value from the other, and compares the absolute value of the result with 10- n. If 10- n is greater than the absolute value of the subtraction result, the numbers are equal.

A value of -1 indicates a full comparison.

Example

Compare the values 3.289999 and 3.290000 with a numeric precision of 2:

NUMERICPRECISION 2

Is 3.289999 == 3.290000 given a numeric precision of 2?

| 3.289999 - 3.290000 | = 0.000001 (the absolute value)

10-2 = 0.01

0.01 > 0.000001, so the numbers are equal.