Defining Background Processes

This section describes how to set up the background processes that perform many important functions throughout your product such as:

·         Processing To Do Entries

·         Processes that purge data

·         Processes that archive data

·         And many more…

There are also batch processes that will apply to your specific source application.  Please refer to the documentation section that applies to your source application for more information.

Contents

The Big Picture of Background Processes

Defining Batch Controls

The topics in this section provide background information about a variety of background process issues.

Contents

Background Processing Concepts

Parameters Supplied To Background Processes

Processing Errors

System Background Processes

Submitting Batch Jobs

Parallel Background Processes

How to Re-extract Information

How to Submit Batch Jobs

How to Track Batch Jobs

How to Restart Failed Jobs and Processes

Background Processing Concepts

While the system uses a 3^rd party scheduler to secure and execute its background processes, there are additional issues that you should be familiar with:

·         Batch control records are used for the following purposes:

·         For processes that extract information, the batch control record defines the next batch number to be assigned to new records that are eligible for extraction.  For example, the batch control record associated with the process that extracts bill print information defines the next batch number to be assigned to recently completed bill routings.  When this bill print extract process next runs, it extracts all bill routings marked with the current batch number (and increments the next batch number).

·         The batch control record for each background process organizes audit information about the historical execution of the background process.  The system uses this information to control the restart of failed processes.  You can use this information to view error messages associated with failed runs.

·         Many processes have been designed to run in parallel in order to speed execution.  For example, the process that produces bills in Oracle Utilities Customer Care and Billing can be executed so that bills are produced in multiple ”threads” (and multiple threads can execute at the same time).  Batch control records associated with this type of process organize audit information about each thread in every execution.  The system uses this information to control the restart of failed threads.  Refer to Parallel Background Processes for more information.

·         Some processes define extra parameters.  These parameters are defined with the batch control and will be used when the background process is submitted on-line.

The following diagram illustrates the relationships that exist for batch control records.  This picture is meant for your technical staff.

Results of each batch run can be viewed using the Batch Run Tree page.

Parameters Supplied To Background Processes

All background processes receive the following parameters.

·         Batch code.  Batch code is the unique identifier of the background process.

·         Batch thread number.  Thread number is only used for background processes that can be run in multiple parallel threads.  It contains the relative thread number of the process.  For example, if the billing process has been set up to run in 20 parallel threads, each of the 20 instances receives its relative thread number (1 through 20).  Refer to Optimal Thread Count for Parallel Background Processes for more information.

·         Batch thread count.  Thread count is only used for background processes that can be run in multiple parallel threads.  It contains the total number of parallel threads that have been scheduled.  For example, if the billing process has been set up to run in 20 parallel threads, each of the 20 instances receives a thread count of 20.  Refer to Optimal Thread Count for Parallel Background Processes for more information.

·         Batch rerun number.  Rerun number is only used for background processes that download information that belongs to given run number.  It should only be supplied if you need to download an historical run (rather than the latest run).

·         Batch business date.  Business date is only used for background processes that use the current date in their processing.  For example, billing using the business date to determine which bill cycles should be downloaded.  If this parameter is left blank, the system date is used.  If supplied, this date must be in the format YYYY-MM-DD.  Note: this parameter is only used during QA to test how processes behave over time.

·         Override maximum records between commits.  This parameter is optional and overrides each background process’s Standard Commit.  You would reduce this value, for example, if you were submitting a job during the day and you wanted more frequent commits to release held resources.  You might want to increase this value when a background process is executed at night (or weekends) and you have a lot of memory on your servers.

·         Override maximum minutes between cursor re-initiation.  This parameter is optional and overrides each background process’s Standard Cursor Re-Initiation Minutes.  You would reduce these values, for example, if you were submitting a job during the day and you wanted more frequent commits to release held resources (or more frequent cursor initiations).  You might want to increase these values when a background process is executed at night (or weekends) and you have a lot of memory on your servers.

·         User ID.  Please be aware of the following in respect of user ID:

·         The user ID is a user who should have access to all application services in the system.  This is because some batch processes call application services to perform maintenance functions (e.g., when an account is updated, the batch process may call the account maintenance application service). 

·         Some batch processes stamp a user ID on records they create / update.  For example, the billing process stamps this user ID on financial transactions it creates.

·         This user ID’s display profile controls how dates and currency values are formatted in messages and bill calculation lines.

·         Password.  Password is not currently used.

·         Language Code.  Language code is used to access language-specific control table values.  For example, error messages are presented in this language code.

·         Trace program at start (Y/N), trace program exit (Y/N), trace SQL (Y/N) and output trace (Y/N).  These switches are only used during QA and benchmarking.  If trace program start is set to Y, a message is displayed whenever a program is started.  If trace program at exist is set to Y, a message is displayed whenever a program is exited.  If trace SQL is set to Y, a message is displayed whenever an SQL statement is executed.  If output trace is set to Y, special messages formatted by the background process are written.

Contents

Override Maximum Errors in Batch Process Parameter

Extra Parameters

Override Maximum Errors in Batch Process Parameter

Each of the batch processes has, as part of its run parameters, a preset constant that determines how many errors that batch process may encounter before it is required to abort the run. A user can override this constant with an optional additional parameter (MAX-ERRORS). If a user chooses not to enter a value for the parameter, the process uses its own preset constant.

The input value must be an integer that is greater than or equal to zero. The maximum valid value for this parameter is 999,999,999,999,999.

The syntax for entering this additional parameter when submitting the batch process is “MAX-ERRORS=PARM VALUE”, where the PARM VALUE is the desired value (e.g., MAX-ERRORS=50). 

Extra Parameters

A limited number of background processes receive additional parameters.  When a process receives additional parameters, they are documented adjacent to the respective batch process description in the following sections in the Extra Parameters column.

The syntax for entering these parameters when submitting the batch process is “PARM-NAME=PARM VALUE”, where PARM-NAME is the name of the parameter as it is documented below for each batch process and the PARM VALUE is the desired value.  For example, if the documentation indicates that the extra parameter for a particular batch process is ADD-WORK-DAYS, with possible values of Y and N, and you want to pass a value of N, enter the following when prompted: ADD-WORK-DAYS=N.

Indicating a File Path

Some of the system background processes use extra parameters to indicate a File Path and/or File Name for an input file or an output file.  For example, most extract processes use File Path and File Name parameter to indicate where to place the output file.

When supplying a FILE-PATH variable, the directory specified in the FILE-PATH must already exist and must grant write access to the administrator account for the product.  You may need to verify a proper location with your systems administrator.

The syntax of the FILE-PATH depends on the platform used for the product application server.  Contact your system administrator for verification.  For example, if the platform is UNIX, use forward slashes and be sure to put a trailing slash, for example /spltemp/filepath/.

Processing Errors

When a background process detects an error, the error may or may not be related to a specific object that is being processed.  For example, if the program finds an error during batch parameter validation, this error is not object-specific.  However, if the program finds an error while processing a specific bill, this error is object-specific.  The system reports errors in one of the following ways:

·         Errors that are not object-specific are written to the error message log in the Batch Run Tree.

·         Some batch processes create entries in an “exception table” for certain object-specific errors.  For example, an error detected in the creation of a bill in Oracle Utilities Customer Care and Billing may be written to the bill exception table.  If an error is written to an exception table, it does not appear in the batch run tree.  For each exception table, there is an associated to do entry process that creates a To Do Entry for each error to allow a user to correct the problem on-line.

·         For some background processes, errors that do not result in the creation of an exception record may instead generate a To Do entry directly.  For these processes, if you wish the system to directly create a To Do entry, you must configure the To Do type appropriately.  Refer to To Do entry for object-specific errors for information about configuring the To Do type.  If the background process detects an object specific error AND you have configured the system to create a To Do entry, the error is not written to the batch run tree.  If you have configured your To Do type to not create To Do entries for certain errors, these errors are written to the batch run tree.

System Background Processes

Submitting Batch Jobs

Most batch jobs are submitted via a batch scheduler.  Your organization may use a third party batch scheduler or may use the batch scheduling functionality provided with the system.

In addition, you can manually submit your adhoc background processes or submit a special run for one of your scheduled background processes using the on-line batch job submission page.

Technical Implementation Of Online Batch Submission

This section provides information about batch jobs are processed via the online batch submission.

Contents

A PERL Program Determines If There's Work To Do

A Common Routine Is Invoked When Batch Programs Complete

Polling Frequency

A PERL Program Determines If There's Work To Do

A PERL program runs in the background looking for Pending batch jobs and then executes them.  When a batch job is executed, a batch run is created.  The batch run keeps track of the overall status of the various parallel threads and it manages restart logic should the run fail.  The following flowchart provides a schematic of this logic:

When a Pending batch job is detected by the PERL program, the following takes place:

·         The batch job's status is changed to Started.

·         y parallel threads of the corresponding batch program are executed.  Refer to Running Multi-threaded Processes for more information.

When the batch program executes, its first thread has the responsibility of creating an In Progress batch run (assuming this is not a restarted run). 

A Common Routine Is Invoked When Batch Programs Complete

When all threads of a batch program complete, a common routine is called.  The following flowchart provides a schematic of this routine's logic:

The following points summarize important concepts illustrated in the flowchart:

·         The batch job's status is changed to Ended.

·         The system checks if all threads were successful:

·         If at least one thread fails,

·         An email is sent (if an email address is defined on the batch job request).

·         The batch run's status is changed to Error.

·         If all threads are successful,

·         An email is sent (if an email address is defined on the batch job request).

·         The batch run's status is changed to Complete.

Polling Frequency

Usage of online batch submission requires that your system administrator create a job in the operating system to periodically execute the PERL program described above.  The topics in this section describe how to do this for the Windows and Unix operating systems.

Contents

Windows Operating System

Unix Operating System

Windows Operating System

The configuration of the Periodic polling under Windows is documented in the Windows installation guide in the “Configuring Batch Submission to Run Via a Job Scheduler” section.

Unix Operating System

The configuration of the Periodic polling under UNIX is documented in the installation guide in the “Configuring cdxcronbatch.sh to Run Via a Job Scheduler” section.

Parallel Background Processes

Many processes have been designed to run in parallel in order to speed execution.  For example, the billing process in Oracle Utilities Customer Care and Billing can be executed so that bills are produced in multiple ”threads” (and multiple threads can execute at the same time).

By default the system distributes the data for the multiple threads using the primary key of the main table of the process.    For example, if you are running BILLING in Oracle Utilities Customer Care and Billing with multiple threads, the batch process distributes the accounts across the multiple threads.  The Account ID is 10 digits long so if you run it with 4 threads, the records are processed as follows:

·         Thread 1: Account IDs 0000000001 - 2500000000

·         Thread 2: Account IDs 2500000001 - 5000000000

·         Thread 3: Account IDs 5000000001 - 7500000000

·         Thread 4: Account Ids 7500000001 - 9999999999

Note that the multi-threading logic relies on the fact that primary keys for master and transaction data are typically system generated random keys.

Optimal Thread Count

Running a background process in multiple threads is almost always faster than running it in a single thread.  The trick is determining the number of threads that is optimal for each process. 

This is a rule of thumb because each process is different and is dependent on the data in your database.  Also, your hardware configuration (i.e., number of processors, speed of your disk drives, speed of the network between the database server and the application server) has an impact on the optimal number of threads.  Please follow these guidelines to determine the optimal number of threads for each background process:

·         Execute the background process using the number of threads dictated by the rule of thumb (described above).  During this execution, monitor the utilization percentage of your application server, database server and network traffic. 

·         If you find that your database server has hit 100% utilization, but your application server hasn’t one of the following is probably occurring:

·         There may be a problematic SQL statement executing during the process.  You must capture a database trace to identify the problem SQL.

·         It is also possible that your commit frequency may be too large.  Commit frequency is a parameter supplied to every background process.  If it is too large, the database’s hold queues can start swapping.  Refer to Parameters Supplied to Background Processes for more information about this parameter.

·         It is normal if you find that your application server has hit 100% utilization but your database server has not.  This is normal because, in general, all processes are CPU bound and not IO bound.  At this point, you should decrease the number of threads until just under 100% of the application server utilization is achieved.  And this will be the optimal number of threads required for this background process. 

·         If you find that your application server has NOT hit 100% utilization, you should increase the number of threads until you achieve just under 100% utilization on the application server.  And remember, the application server should achieve 100% utilization before the database server reaches 100% utilization.  If this proves not to be true, something is probably wrong with an SQL statement and you must capture an SQL trace to determine the culprit.

Another way to achieve similar results is to start out with a small number of threads and increase the number of threads until you have maximized throughput.  The definition of “throughput” may differ for each process but can be generalized as a simple count of the records processed in the batch run tree.  For example, in the Billing background process in Oracle Utilities Customer Care and Billing, throughput is the number of bills processed per minute.  If you opt to use this method, we recommend you graph a curve of throughput vs. number of threads.  The graph should display a curve that is steep at first but then flattens as more threads are added.  Eventually adding more threads will cause the throughput to decline.  Through this type of analysis you can determine the optimum number of threads to execute for any given process.

How to Re-extract Information

If you need to recreate the records associated with an historical execution of an extract process, you can – simply supply the desired batch number when you request the batch process.

How to Submit Batch Jobs

You can manually submit your adhoc background processes or submit a special run for one of your scheduled background processes. 

How to Track Batch Jobs

You can track batch jobs using the batch process pages, which show the execution status of batch processes.  For a specified batch control id and run id, the tree shows each thread, the run-instances of each thread, and any messages (informational, warnings, and errors) that might have occurred during the run. 

How to Restart Failed Jobs and Processes

Every process in the system can be easily restarted if it fails (after fixing the cause of the failure).  All you have to do is resubmit the failed job; the system handles the restart.

The system is delivered with all necessary batch controls.  If you introduce a new background process, open Admin Menu, Batch Control to define the related batch control record.  Refer to Background Processing Concepts for more information.

Description of Page

Enter an easily recognizable Batch Process and Description for each batch process.

Owner indicates if this batch control is owned by the base package or by your implementation (Customer Modification).  The system sets the owner to Customer Modification when you add a batch control.  This information is display-only.

Use the Detailed Description to describe the functionality of the batch process in detail.

Use Program Type to define if the batch process program is written in Java or COBOL.

Use Program Name to define the program associated with your batch process:

·         If the Program Type is COBOL, enter the name of the COBOL program.

·         If the Program Type is Java, enter the Java class name.

The Last Update Timestamp, Last Update Instance and Next Batch Nbr are used for audit purposes.

Turn on Accumulate All Instances to control how this batch control is displayed in the Batch Run Tree.  If checked, the run statistics (i.e., "Records Processed" and "Records in Error") for a thread are to be accumulated from all the instances of the thread.  This would include the original thread instance, as well as any restarted instances.  If this is not turned on, only the ending (last) thread instance's statistics are used as the thread's statistics.  This may be preferred for certain types of batch processes where the accumulation would create inaccurate thread statistics, such as those that process flat files and, therefore, always start at the beginning, even in the case of a restart.

The parameter collection is used to define additional parameters required for a particular background process.  The following fields should be defined for each parameter:

Sequence                                            Defines the relative position of the parameter.

Parameter Name                                 The name of the parameter as defined in System Background Processes.

Description                                          A description of the parameter.

Detailed Description                            A more detailed description of the parameter.

Required                                              Indicate whether or not this is a required parameter.

Parameter Value                                 Enter a default value, if applicable.

Owner                                                  Indicates if this batch process is owned by the base package or by your implementation (Customer Modification).  The system sets the owner to Customer Modification when you add a batch process.  This information is display-only.