Tuning SQR Numerics

SQR for PeopleSoft provides three types of numeric values:

• Machine floating point numbers

• Decimal numbers

• Integers

Machine floating point numbers are the default. They use the floating point arithmetic that is provided by the hardware. This method is very fast. It uses binary floating point and normally holds up to 15 digits of precision.

Some accuracy can be lost when you are converting decimal fractions to binary floating point numbers. To overcome this loss of accuracy, you can sometimes use the ROUND option of commands such as ADD, SUBTRACT, MULTIPLY, and DIVIDE. You can also use the round function of LET or numeric edit masks that round the results to the needed precision.

Decimal numbers provide exact math and precision of up to 38 digits. Math is performed in the software. This is the most accurate method, but also the slowest.

You can use integers for numbers that are known to be integers. Using integers is beneficial because they:

• Enforce the integer type by not allowing fractions.

• Adhere to integer rules when dividing numbers.

Integer math is also the fastest method, typically faster than floating point numbers.

If you use the DECLARE-VARIABLE command, the -DNT command-line flag, or the DEFAULT-NUMERIC entry in the Default-Settings section of the PSSQR.INI file, you can select the type of numbers that SQR uses. Moreover, you can select the type for individual variables in the program with the DECLARE-VARIABLE command. When you select decimal numbers, you can also specify the needed precision.

Selecting the numeric type for variables enables you to fine-tune the precision of numbers in your program. For most applications, however, this type of tuning does not yield a significant performance improvement, so selecting decimal is best. The default is machine floating point to provide compatibility with older releases of the product.