13 Batch Processes and RPAS Utilities

Included with an RPAS installation is a collection of stand-alone executables and scripts that are used for a variety of operations. RPAS utilities are run directly against a domain. In a global domain environment, most utilities can only be run on the master domain. RPAS utilities can be categorized into the following groupings:

• Hierarchy management: The loading and refreshing of hierarchies, and the process of updating the data structures in the domain to reflect hierarchy changes

• Measure data: Utilities for loading, exporting, and moving data within and between domains

• Miscellaneous: A variety of utilities for performing certain procedures in batch and for setting a number of parameters on an environment or a domain

• Information RPAS utilities: A variety of utilities that retrieve information about a domain, data, the RPAS Server code, or an object used by the server

This chapter contains the following sections:

• CSV File Format

• RPAS Utilities Logging Options

• Using Shell Scripts to Run Batch Processes

• Common Information and Parameters for RPAS Utilities

• RPAS Intraday Enabler

CSV File Format

For those utilities that use a comma-separated value (CSV) file, the following formatting applies for any commas or double quotation marks in the data:

• If the data does not contain any commas or double quotation marks, it does not need any special formatting.

• If the data contains a comma, the string must be enclosed between opening and closing double quotation marks.

• If the data contains quotation marks, the string must be enclosed between opening and closing double quotation marks and any embedded quotation marks must be paired.

Table 13-1 shows examples of the formatting.

Table 13-1 CVS File Format

Data	Formatted Data
Item 001	Item 001
Item 001, Soda	"Item 001, Soda"
"Large Screen" TV	""Large Screen"" TV"
Item 002, "Generic Brand" Cereal	"Item 002, ""Generic Brand"" Cereal"

RPAS Utilities Logging Options

RPAS has a number of applications used to control or process data. Currently there are no unified methods for logging output, controlling the level of logging, or directing logging to a particular file. Instead each utility has its own methods, although many are similar. The current behavior for each utility follows.

Log Levels

The standard log levels are controlled by the -loglevel option. Not all programs use these levels, but most do. The default logging level is Warning, which means that any log messages that are specified as a warning or higher will be output.

• All: Forces all log levels to be output

• Profile: Performance profiling information

• Audit: User-specific domain and workbook activities. These activities include the following:

  • Workbook build, calculation, save, commit, and custom menu operations

  • User login and logout to the domain

• Information: General status messages that are not problematic. Outputs status and progress of the operation, in addition to the error and warning messages.

• Warning: Messages indicating a potential problem, but not one that is fatal. Outputs warning messages, in addition to error messages.

• Error: Messages relating to a fatal problem. Outputs only error messages.

• None: No messages. No output is provided if the utility successfully executes.

Each of the lines that contain the above types of feedback is normally preceded with a code that indicates what type of information is being output. Each code should have an angle bracket (<) in front of it.

• E indicates that the message is an error.

• W indicates that the message is a warning.

• I indicates that the message is informational.

• U indicates that the message is audit-relevant information.

• P indicates that the message is a performance profile.

Note: Audit information related to workbook activities is recorded in rpas.log under each user's working directory. Information related to domain activities, such as user sign-on and sign-off, is recorded in DomainDaemon.log.

Utilities with Standard Logging

A number of utilities allow for the -loglevel option to control which messages are output to the screen. There is no way to log to a file directly. Table 13-2 displays the utilities that can use the -loglevel option.

Table 13-2 Utilities That Can Use the -loglevel Option

alertmgr	regfunction
checkDomain	regmeasattr
configCommitAsap	regtemplate
createdb	regTokenMeasure
createGlobalDomain	reguserdim
dattrmgr	rpasverison
dbdiff	rtkappcnfgmeas
dimensionMgr	showWorkbookQueues
domaininfo	syncNAValue
ldrule	updateArray
listDb	updatestyles
mapData	upgradeDomain
moveDomain	usermgr
printArray	wbmgr
printMeasure

Scripts

Shell scripts cannot use standard logging, but may execute the following programs that use it:

convertDomain

All output to the screen.

createRpasDomain

The -v option controls the type of messages sent to the screen.

Utilities with Multi-Process Logging

Some utilities are based on the multi-processes domain utility framework. These utilities send messages to the screen and a log file master.log. Any child processes output messages to a log file in the domain/output directory named subdomain0000.log where the number indicates the subdomain being processed. This directory will contain all log files created during the run of that utility. This change has been updated so that the controlling process logs to the screen as well as to a file in that directory. The newly created directory name is formatted as APPNAMEYYYYMMDDHHMIbXX, where APPNAME is the utility name, YYYY is the year, MM is the month, DD is the day, HH is the hour, MI is the minute. The character b and XX (two digits) are used to make the directory name unique. The framework attempts to limit the number of directories created for any single utility to eight. The parameter -loglevel can be used to control the type of messages sent to the screen and log file.

These utilities are as follows:

• exportData

• loadmeasure

• loadHier

• exportMeasure

• reconfigGlobalDomainPartitions

• wbbatch

• reindexDomain

domainprop

The domainprop utility only provides logging to the screen.

hierarchyMgr

The hierarchyMgr utility only provides logging to the screen.

configCommitAsap

This utility should be started from the RpasDbServer application when the client requests a workbook to be committed.

Utilities with Special Logging

These utilities may use standard logging with additional features or may use entirely different logging methods.

DomainDaemon

The DomainDaemon uses standard logging. It logs output to a file (see below). The file is created either in the current working directory or in the directory specified by the RPAS_LOG_PATH environment variable.

The file name depends on the RPAS_LOG_BACKUPS environment variable. If it is set to 1 or greater, then:

The log file name is Daemon_Dyyyymmddhhmmbxx.log where yyyy is the current year, mm is the current month, dd is the current day, hh is the current hour, mm is the current minute. The character b and xx (a two-digit number) are used to make the file name unique.

The number of these log files is limited to the number provided in the environmental variable RPAS_LOG_BACKUPS.

Otherwise, the log file name is Daemon.log. Any existing log file is renamed to Daemon.old.

At midnight, the current log file is closed and a new one opened, named as described above.

RpasDbServer

This should only be started from the DomainDaemon as a part of a client request to start an RPAS session. The logging level is controlled by the client's RPAS_LOG_LEVEL environment variable. If it is not set, then it defaults to logging messages at the warning level.

This utility creates log files in the domain/users/client directory, where the domain is the current domain path and the client is the current client. The actual file name used is either rpas Dyyyymmddhhmmbxx.log or rpas.log, based on the environmental variable RPAS_LOG_BACKUPS (c.f., DomainDaemon, above).

loadHier

The loadHier utility uses standard logging. This utility performs part of its processing in child processes. loadHier provides a list of all hierarchy positions that have been changed since the previous hierarchy load. The resulting directory name is:

<utility><YYMMDDHHMISS><pXXXXX><bYY>

where utility is the name of the program (for example,-loadmeasure), followed by a time and date stamp, then the process id (pXXXX), and then the character b and a 2-digit number to avoid conflicts (bYY).

If any problems occur when loading specific records that belong to the partition hierarchy, they are reported in the format as shown below. Note that the record is completely reproduced in this error report in the log.

<E 2008Jul02 12:04:52.196> Could not find position '90000044' in line number 3:
'2001052090000044 1000 7 '. Skipping!

Problems with records along non-partitioned hierarchies are reported as shown in the following example:

<I 2008Jul02 12:04:55.482> MeasureLoader::loadDataFromFile() Loading '.ovr' file '/vol.nas/u09/rpasqc/qc_testing/aix/1208rc2_test/RDF_12/ldom1/input/psal.ovr'
<D 2008Jul02 12:04:55.514> Error on line 1: '2001031110000044 STR1000 8 ' .Position name: STR_STR1000 not found.
<D 2008Jul02 12:04:55.514> Error on line 2: '2011041510000044 1000 9 ' .Position name: DAY20110415 not found.
<D 2008Jul02 12:04:55.964> 2 lines had problematic data.

locked

Messages are sent only to the screen.

mace

The mace utility uses standard logging. The -debugRuleEngine option logs some messages to the file mace.log in the current working directory.

reconfigGlobalDomainPartitions

Uses standard logging. The loadHier process may be spawned as a child process. See "loadHier" for additional details. When the loadHier utility is started as a child process it remaps the screen output of to the log file loadHier.log contained in the current working directory.

renamePositions

Uses standard logging. The -log option overwrites the default log file name of hierName and Rename.log in the current working directory. The -loglevel parameter does not control the types of messages written to this log file.

regmeasureServer

This application should only be started from the RPAS libraries to process measure registration and de-registration. Each process creates a log file in a newly created directory in the domain output directory. The newly created directory name is formatted as regServerYYYYMMDDHHMIbXX, where YYYY is the year, MM is the month, DD is the day, HH is the hour, MI is the minute. The character b and XX (two digits) are used to make the directory name unique. The RPAS libraries attempt to create at most eight directories for any single application.

Utilities and Database Locks

On UNIX and Linux platforms, before running any RPAS program, it is important to run "umask 0". Otherwise the RPAS databases may be permanently locked out to access by other users or groups. This applies to any program that accesses any RPAS database, including, but not limited to:

• convertDomain

• createRpasDomain

• DomainDaemon

• domaininfo

• exportMeasure

• ldrule

• listDb

• loadhier

• loadmeasure

• mace

• printArray

• printMeasure

• printMeasureInfo

• regmeasure

• rpasInstall

• RPASODBCAgent

• RPASODBCDataService

• usermgr

If database lock files (*.db.lck) have been created with limited permissions that block other users' access, a workaround is to run the following command when logged in as the lock-file-owning user:

$ find {domainPath} -user {owner} -name \*.lck -exec rm -f {} ';'

where {domainPath} is the path to the domain and {owner} is the UNIX user that owns the lock files.

Using Shell Scripts to Run Batch Processes

Batch processes should be written using scripts that call the RPAS 11 binaries found in the $RPAS_HOME/bin/ directory. Any log files generated by scripts are in the [DOM]/scripts/err/ directory. Examples of tools include Korn Shell, Python, and Perl.

A Sample Shell Script

The following sample shell script loads the product and location hierarchies into a domain. It is assumed that this script is invoked from the [DOM]/scripts/ directory.

1  #!/bin/ksh
2 loadHier -d .. -load prod > ./err/loadhier.prod.log
3  loadHier -d .. -load loc >> ./err/loadhier.loc.log

Line 1 defines the shell that executes the script. In this example, it is defined to be the Korn shell. Therefore, this script is always executed from the Korn shell, even if the user's shell is different.

Lines 2 and 3 call the loadHier utility to load the latest product and location hierarchy information. Depending on the batch process to be performed by the shell script, lines 2 and 3 can be replaced by one or more lines to call one or more RPAS utilities.

Common Information and Parameters for RPAS Utilities

A number of standard arguments are available for most RPAS utilities. Check the usage of a specific utility to verify whether or not it is available.

Table 13-3 Standard Arguments for RPAS Utilities

Argument	Description
-version	Obtains the version information of the utility (for instance, RPAS 13.2.0). It does not require -d domainPath.
-d pathtodomain	Specifies the path to the domain against which the utility will run or from which data will be used.
-loglevel	See "RPAS Utilities Logging Options" for more information.
-n	Certain utilities contain this parameter to perform a dry run. Use this option to see what would change without making actual changes to the system or data.See the usage of a specific utility to see whether this option is applicable.
-noheader	Disables the use of a timestamp in the header of the log file.
-help-?-usage	Any of these arguments output the utility information and syntax to the terminal window. This can also be accomplished by running the utility with no arguments.

Logger verbosity levels determine how much information is generated on the terminal for a given utility. An administrator can set these levels for each RPAS utility. The available logger verbosity levels are as follows:

• none: No output if the utility successfully executes

• error: Outputs only error messages

• warning: Outputs warnings in addition to error messages

• information: Outputs status and progress of the operation in addition to the error and warning messages

• all: Outputs all available information generated by the utility, including error, warning, and informational messages

Each line, which contains the above type of feedback, is normally preceded with a code that indicates what type of information is being output. Each code should have an angle bracket (<) in front of it. E indicates the message is an error. W indicates the message is a warning. I indicates the message is informational.

Configuration Tools Log Files

For the RPAS Configuration Tools, information is logged in the files stderr.txt and stdout.txt, which are located in the bin subdirectory of the Tools directory. If a problem with the configuration tools is encountered, send these two files to Oracle Retail Customer Care, along with a description of the problem.

RPAS Intraday Enabler

The RPAS Intraday Enabler (ride) functionality enables batch operations to be run over an RPAS domain while users are accessing workbooks and completing workbook operations.

This functionality enables batch operations to be executed over a domain, but does not prevent users from accessing other components that do not affect or interfere with the batch operations. The running of an exclusive batch process does not cause any pre-existing workbook operations that require domain access to fail or terminate. Users in domains that are not part of the exclusive process are not affected in any way.

In domains that have been locked by an exclusive batch process, the users are still able to perform operations that only require access to the workbook. The operations include the following:

• Workbook Edits

• Workbook Calculations

• Workbook Saves

• Workbook Opens

• Workbook Navigation

• Commit ASAP Entry

• Commit Later Entry

Workbooks entered into the commit ASAP queue while an exclusive lock is in place are processed after the exclusive process is complete.

In these same domains, users cannot perform operations that require access to data within the domain. The access can be either read or write. The operations that are prevented include the following:

• Workbook Build

• Workbook Refresh

• Workbook Custom Menu (unless configured as intraday-concurrent)

• Workbook Commit Now/Later

• Insert Measure

• Dynamic Position Maintenance (DPM)

When users try to access one of these operations after the exclusive lock is obtained, they see a message stating that an exclusive process is running. A default message is provided, which can be replaced with a message as part of the call to the ride utility.

When users are working with a workbook in the master domain, a lock is required on the master domain and all local domains that are needed for the operation. The workbook operations in this domain are blocked when the master or local domain data is accessed by the ride process. See "Scenarios" for more details on domain access during different types of ride processes.

Configuration functionality is provided so that a custom menu can be flagged to run concurrently with a ride process (intraday-concurrent).

Note: See the RPAS Configuration Tools User Guide for details on how to configure a custom menu to be intra-day-concurrent. A custom menu that is configured to run concurrently with a ride process should only access workbook data, run a script that uses the ride utility, and run commits using the commit ASAP functionality. Custom menus that update or read directly from the domain should not be configured as intra-day-concurrent as this can conflict with the ride process.

In order for a batch job to run over a domain without interference from an online activity, exclusive domain access must be granted to the job that is running. This is achieved by creating a domain access control using a dual-lock control. The domain access control manages the lock request from workbooks and ride processes.

Figure 13-1 Domain Access Control

Figure 13-1 shows the process control that is in place with the locking schema. In this case, an administrator requests exclusive access to a domain in order to run a batch job. This requires an exclusive lock on the domain so that the job can run. After the lock is received, no other workbook operations can obtain write access to the domain until the process is complete. If the exclusive lock cannot be obtained, the process should time out and the administrator notified based on the output of the ride utility. When the ride utility times out, the domainStatus utility is automatically run to provide details of the user workbook operations that are blocking the ride process. See the "Domain Lock Status Using domainStatus" section on for details on that procedure.

ride Usage

ride -d domain -process pname|-script sname -args args {-message messageString} {-timeout minutes} {-wait minutes} {-partitions pos1,pos2,…} {-masterInBatch}

Table 13-4 provides descriptions of the arguments used by the ride utility.

Table 13-4 ride Utility Arguments

Argument	Description
-d domain	Refers to a simple or master domain. When the master domain is specified, all local domains in the global domain environment, as well as the master domain, are locked.
-process pname	The name of the process to execute. This parameter cannot be used with the -script parameter.
-script sname	The name of the script to execute. This parameter cannot be used with the -process parameter.
-args args	Process arguments passed to the script or process to be executed. The -args parameter must be the last parameter or switch for this application. All parameters or switches after the -args parameter are passed on to the process or script to be started.
-message messageString	This optional override message is presented to the user who is trying to perform an operation blocked by an intra-day batch process. A default message is provided to the user if this argument is not provided.
-timeout minutes	The utility will time out if it cannot get access to the domains during this time. By default, there is no timeout. The timeout starts when the control utility is executed.
-wait minutes	The time to wait before starting the process or script. Even if domain access is granted, the process does not start until the end of wait time. The clock starts when the control utility is executed. The default is 0.
-partitions pos1,pos2,…	Partition positions (such as dept1, dept2, and so on) that determine the local domains that are accessed by the process or script.
-masterInBatch	When running over a global domain environment, the master domain is accessed by the process or script in addition to any local domains selected.

Note: If neither -partitions nor -masterInBatch are provided on the command line, the entire domain will be processed when running over a global domain environment (that is, all subdomains and the master).

Scenarios

This section provides several scenarios that are possible with the ride utility. The scenarios explain the domain access based on how the ride utility is executed. The tables indicate whether the specific operations are blocked or allowed.

Scenario 1

This scenario describes running ride and specifying only the master domain. This locks all domains.

Usage: ride -d master -script script.ksh

Table 13-5 Running the ride Utility Specifying Only the Master Domain

Workbook Operation		Master Domain	Local Domain 1	Local Domain 2	Local Domain 3
Build	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Blocked
Refresh	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Blocked
Commit Now	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Blocked
Custom Menu (not ride concurrent)	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Blocked
Insert Measure	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Blocked
DPM		NA	Blocked	Blocked	Blocked

Scenario 2

This scenario describes running ride and specifying only one local domain (local domain with partition position 100).

Usage: ride -d master -script script.ksh -partitions 100

Table 13-6 Running the ride Utility Specifying One Local Domain

Workbook Operation		Master Domain	Local Domain 1	Local Domain 2	Local Domain 3
Build	HBI Measures	Blocked	Blocked	Allowed	Allowed
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Refresh	HBI Measures	Blocked	Blocked	Allowed	Allowed
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Commit Now	HBI Measures	Blocked	Blocked	Allowed	Allowed
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Custom Menu (not ride concurrent)	HBI Measures	Blocked	Blocked	Allowed	Allowed
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Insert Measure	HBI Measures	Blocked	Blocked	Allowed	Allowed
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
DPM		NA	Blocked	Allowed	Allowed

Scenario 3

This scenario describes running ride and specifying only one local domain (local domain with partition position 100) and the masterInBatch option.

Usage: ride -d master -script script.ksh -partitions 100 -masterInBatch

Table 13-7 Running the ride Utility Specifying One Local Domain and masterInBatch

Workbook Operation		Master Domain	Local Domain 1	Local Domain 2	Local Domain 3
Build	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Refresh	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Commit Now	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Custom Menu (not ride concurrent)	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
Insert Measure	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Allowed	Allowed
DPM		NA	Blocked	Blocked	Blocked

Scenario 4

This scenario describes running ride and specifying two local domains (local domain 1 with partition position 100 and local domain 2 with partition position 200).

Usage: ride -d master -script script.ksh -partitions 100,200

Table 13-8 Running the ride Utility Specifying Two Local Domains

Workbook Operation		Master Domain	Local Domain 1	Local Domain 2	Local Domain 3
Build	HBI Measures	Blocked	Blocked	Blocked	Allowed
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Refresh	HBI Measures	Blocked	Blocked	Blocked	Allowed
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Commit Now	HBI Measures	Blocked	Blocked	Blocked	Allowed
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Custom Menu (not ride concurrent)	HBI Measures	Blocked	Blocked	Blocked	Allowed
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Insert Measure	HBI Measures	Blocked	Blocked	Blocked	Allowed
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
DPM		NA	Blocked	Blocked	Allowed

Scenario 5

This scenario describes running ride and specifying two local domains (local domain 1 with partition position 100 and local domain 2 with partition position 200) and the masterInBatch option.

Usage: ride -d master -script script.ksh -partitions 100,200 -masterInBatch

Table 13-9 Running the ride Utility Specifying Two Local Domains and the masterInBatch Option

Workbook Operation		Master Domain	Local Domain 1	Local Domain 2	Local Domain 3
Build	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Refresh	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Commit Now	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Custom Menu (not ride concurrent)	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
Insert Measure	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Blocked	Blocked	Allowed
DPM		NA	Blocked	Blocked	Blocked

Scenario 6

This scenario describes running ride to lock the master domain.

Usage: ride -d master -script script.ksh -masterInBatch

Table 13-10 Running the ride Utility to Lock the Master Domain

Workbook Operation		Master Domain	Local Domain 1	Local Domain 2	Local Domain 3
Build	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Allowed	Allowed	Allowed
Refresh	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Allowed	Allowed	Allowed
Commit Now	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Allowed	Allowed	Allowed
Custom Menu (not ride concurrent)	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Allowed	Allowed	Allowed
Insert Measure	HBI Measures	Blocked	Blocked	Blocked	Blocked
	No HBI Measures	Blocked	Allowed	Allowed	Allowed
DPM		NA	Blocked	Blocked	Blocked

Domain Lock Status Using domainStatus

The domainStatus utility provides a report on the processes that are locking the domains. The purpose is to identify the user activities that are preventing a ride process from running. This information is important since the ride process does not terminate any existing workbook operation. The utility provides output that includes the process ID, user ID, operation type, and operation start time.

With the output of this utility, the system administrator can determine why the ride process is not running. This should provide enough information to either identify the user who is causing a delay or to manually terminate the process. That will be driven by the specific client and their processes.

Note that this utility reports on domain-level locks used during the ride process. Low-level data locks are not exposed by this utility.

domainStatus Usage

domainStatus -d domain -autoRefresh refreshPeriod

Table 13-11 provides descriptions of the arguments used by the domainStatus utility.

Table 13-11 domainStatus Usage

Argument	Description
-d domain	Refers to a simple or master domain. When master domain is specified, all local domains in the global domain environment are locked as well as the master domain.
-autoRefresh refreshPeriod	Refreshes the lock status information every number of seconds specified by the refreshPeriod.