Batch Concepts
Note: The DISTRIBUTED mode, supported in previous versions of Oracle Utilities Work and Asset Management, had been removed. Customers using this mode should migrate to CLUSTERED mode using either a single server configuration (ss) or unicast configuration (wka) using bedit (see Configuring Batch Using Bedit).
The batch component of the architecture has a number of key concepts to understand.
Batch Clustering Concepts
As mentioned in the previous section, Oracle Coherence is used internally to cluster the resources needed for the batch architecture. This is similar in concept to a cluster used by the JEE web application server for online and web services.
• A cluster is one or more hosts that can execute batch processes. A cluster is named to delineate it from other clusters in your network. Oracle Coherence Named Caches are used to implement the cluster.
• Members of a cluster are hosts, threadpools and submitters. Each member is tracked by the cluster for high availability. At least one member must be active for a cluster to operate.
• Intracluster communication between members of the cluster can be achieved using uni-cast (also known as Well Known Addresses) or multi-cast.
• In uni-cast implementations, each host and associated port is defined in the cluster. Each cluster member has definitions of other members. This mode is a less dynamic configuration but optimizes host to host communications for large clusters.
Note: As each member must have its own configuration. To add or remove members requires manual synchronization of configuration files.
Note: Large clusters are many hosts, large number of threadpools and/or large number of threads executing at the same time.
• In multi-cast (default) implementations, a multi-cast address and port are defined across the cluster which members attach to when executing. This mode is dynamic but traffic across nodes can be excessive for large clusters.
• A threadpool is a Java Virtual Machine (JVM) that executes batch threads. It is a long running JVM that batch threads are submitted to. Threadpools are named to group resources across a cluster when using multiple instances of a threadpool.
• A threadpool has a configurable maximum capacity for running batch threads to prevent overloading of JVM's. Running multiple instances of a threadpool (within or across hosts) can increase the limit and implement high availability.
Note: Threads in background processing are typically heavier in terms of resource usage than typical Java threads hence lower thread limits.
• Clusters can support dedicated cache threadpools. In a cluster, each member can potentially communicate to each other member. In a large network this can cause network overheads that can adversely affect performance. To minimize the impact of network traffic, it is possible to configure a cache threadpool that will act as a central coordinator in the cluster to optimize and channel intramember communication.
• Clusters can support administration threadpools. In a cluster, it is possible to check on the status in real time using JMX across the cluster by connecting to any node in it. It is possible to set up an administration node that can act as a single point of contact across the cluster to monitor performance, in a similar vein as the administration server in the JEE architecture. These adminstration threadpools can be used by command line utilities, jconsole, Oracle Enterprise Manager through Application Management Pack for Oracle Utilities and/or Coherence Management Pack.
• Submitters are instances of individual background processes or individual threads of background processes that are submitted to a named threadpool. The submitter waits for the thread or background process to complete or abort and reports the status of the thread or background process to the operating system. This latter facility is used by Batch Scheduling tools to determine the status of the background process.
• Submitter can be targeted to specific instances of a threadpool or submitted to the cluster of threadpools. Submitters can execute specific threads or all threads in a single execution.
Batch Controls
Note: The system is delivered with all necessary batch controls for the supplied base background processes.
Background processes within Oracle Utilities Work and Asset Management must be defined using meta data (as well as physical code) to execute the process in the architecture. Refer to the online help and user documentation for details of the attributes associated with a batch control.
Standard Parameters
To standardize all the background processes, Oracle Utilities Work and Asset Management uses a number of common standard parameters to uniformly provide functionality across all processes. The table below lists all the standard parameters:
Additional Key Parameters
In all batch controls there are two additional parameters that may not be specified in the batch control definition but are available for use within each job:
• MAX-ERRORS (maxErrors) - This is the error tolerance for the job. When the number of errored records in a thread for a job has been exceeded that thread is stopped with an appropriate error code. This setting is to prevent large amounts of errors to influence the state of the job.
• DIST-THD-POOL (threadPool) - This allows an individual pool to be specified for the execution of the job. This is primarily used with online submission.
• emailMode - Defines the status for which an email notification is sent when a job ends. This does not apply to individual threads but when all threads are complete.
Note: This is used for online submission when an email address is provided on the job submission and the email server is configured.
Valid values are:
• ERROR - Send an email when at least one thread ends in error status.
• SUCCESS - Send an email when all threads end successfully.
• ALL (Default) - Always sends an email when all threads end regardless of status.
• overrideLowIdValue - Specifies a new low id to use in calculating the range for a thread. Oracle Utilities Work and Asset Management assumes by default that the Id is between 0's (for example, 000000000) and 9's (for example, 9999999999), but this parameter will override the low value. The product calculates each thread's low and high id by slicing the job's range into chunks using the thread number and count. It is typically more appropriate for sequentially allocated Ids to evenly distribute work among threads. The parameter value can be an actual number or it can be set to auto. If the parameter is set to auto, then the value is set to the lowest current value on the associated database table. The table name is determined from the batch program's query iterator.
• overrideHighIdValue - Specifies a new high id to use in calculating the range for a thread. Oracle Utilities Work and Asset Management, by default, assumes that the Id is between 0's (for example, 000000000) and 9's (for example, 9999999999), but this parameter will override the high id. The product calculates each thread's low and high id by slicing the job's range into chunks using the thread number and count. It is typically more appropriate for sequentially allocated Ids to evenly distribute work among threads. The parameter value can be an actual number or it can be set to auto. If the parameter is set to auto, then the value is set to the highest current value on the associated database table. The table name is determined from the batch program's query iterator.
• idRangeOverrideClass - Provides for a custom class to do thread range calculation. The class name specified must implement interface com.splwg.base.api.batch.BatchIdRangeOverride. During batch execution, this override class will be instantiated and the setter methods called to initialize the Ids as required. The low and high getter methods will be called to retrieve the high and low ids to be used for the run
• numRecordsToFlush - Defines how frequently to flush the Hibernate cache to prevent high heap consumption and OutOfMemoryErrors. In SingleTransactionStrategy there are no commits throughout the run, so this parameter can be used to flush the Hibernate cache periodically. This parameter is only applicable to jobs that use SingleTransactionStrategy.
Note: The parameters overrideLowIdValue and overrideHighIdValue are only applicable to jobs that use ThreadIterationStrategy.
These parameters may be added to the Batch Control or properties files used for running the job.
Batch Run Tree
Within Oracle Utilities Work and Asset Management’s browser interface, there is an ability to monitor the status and outcomes of individual processes. This can be useful for finding out what actually occurred if an error condition occurred. Refer to the online help for more information about this facility.
The Batch Run Tree is available to any valid user and is a method to communicate the execution information to the relevant business representatives.
Timed Jobs
Note: Monitor processes are ideal for Timed Jobs.
By default, background processes start and finish when they have processed the records it has selected at the start of execution. In this case, to process more records it is necessary to run the batch process again.
In some business processes, it may be necessary to continually execute a job to process work that is ready at any time. For example, processing payments, meter reads, asset changes or tasks as they appear. This situation is handled as a Timed Job. This batch control type is configured with a Timer Interval (the desired runtime between executions in seconds) and a Timer Active flag (to enable and disable the job activity).
When the Timer Active flag is set to true, the product executes the job for the duration of the Timer Interval before executing another instance of the job. This allows the job to run continuously over time, whilst the Timer Active flag is set to true.
Each instance of the job will start and stop using the following rules:
• The job will only execute when the Timer Active flag is set to true. To the next execution from starting this value should set to false.
• The job will execute using the parameters on the Batch Control including the dedicated parameters for Timed Jobs such as UserID, Batch Language and Email Address.
• Once the Timer Interval has been reached, the job will commit after the latest transaction is complete and update the Batch Control is complete or errored (as appropriate).
• The next instance of the process is repeated as above.
Note: Due to varying transaction workloads the time taken may exceed the Timer Interval. This situation can be tracked using Level of Service if desired.
Level of Service
Note: For backward compatibility purposes, Level of Service is in the disabled state and requires configuration of targets in the Level of Service algorithm.
One of the features of the background processing architecture is that all executions of a batch control are tracked internally. This is typically expressed in the Batch Run Tree for information purposes but can also be used to calculate an indicator of the level of service if desired using the Batch Control - Level Of Service algorithm.
Algorithms of this type return one of three levels of service:
• Disabled - Batch level of service algorithm is not enabled for the batch control or no data is available for comparison.
Note:
• Error - Indicates that errors were detected. In this case a message is also returned to indicate the issue.
• Normal - Indicated when there were no errors or issues detected by the algorithm.
Note: It is possible to write custom Level of Service algorithms to compare any appropriate business metric.
Explanation of Commit Interval
The Commit interval parameter is a tuneable parameter to affect the impact of the background processes on the other processes running and prevent internal database errors. In most cases using the defaults will satisfy your site requirements. It is also important to understand their impact to ascertain whether any change is required.
During processing of any background process a main object is used to drive the process. Each process tables one or more objects to process. For example, billing processes accounts, meter reading processes meter reads or meter events, asset Processing uses assets, workforce processing uses tasks etc. For other processes it is other objects that are considered the main object. This status of the object or status of related transaction data is used to determine when a transaction is complete.
For the Commit interval this concept is important as:
• When a certain number of main objects have been processed then a database commit is issued to the database. This number is the expressed in Commit Interval. The larger the commit interval the larger the amount of work that the database has to keep track of between commit points.
Note: The number of records in progress utilizes redo log space so the size of the redo logs in the database and commit interval must be compatible. The higher the commit interval the more redo log space is required.
• At any time in a process a commit for objects processed may be caused by the reaching the Commit Interval or the end of the job, whichever comes first.
Explanation of Thread Limit and Thread Number
One of the features of the Oracle Utilities Application Framework is the ability to run background processes using multiple threads.
The threading concept in Oracle Utilities Work and Asset Management is simple. Each thread takes a predetermined slice of the data to work on. The last thread checks if all other threads are finished and updates the status of the batch control records. For example, if you have 10 threads, then each thread takes 1/10th of the work. As each thread is executing it processes its workload and then completes, the last thread executing is responsible for updating the overall process status to indicate completion.
Implementing threading means you have to execute a number of background processes with an ascending thread number up to the thread limit. For example, if you have a background process with 10 threads, you must run 10 background processes each with a unique thread number between 1 and 10 to complete the background process. Threads can be located on the same machine or different machines. For example, you can run threads 1 to 5 on one machine and threads 6 – 10 on another.
Note: If there is limited data skew in the data then the threads should finish around the same time. If there is some data skew, then some threads may finish later than others.
To implement multi-threading when you submit a process:
• Specify a thread limit greater than 1 as a parameter.
• Execute a background process for every thread with a sequential thread number up to an including the thread limit. There are a couple of implementation guidelines with threading:
• Make sure the number of threads is not excessive. You do not want to flood the CPU’s.
• You must submit a process per thread. In some submission methods this is done automatically and in some it is done manually.
• Threading will increase throughout BUT it will cause higher than usual resource usage (CPU, Disk etc) as well as higher contention. Excessive threading can in fact cause performance degradation in online as well as background processing. Therefore, the number of threads should not be excessive.
Almost all background processes within Oracle Utilities Work and Asset Management support multiple threads (typically the only background processes that are single streamed are extracts and data loads as they involve reading and writing files).
Explanation of Restart and Rerun
Oracle Utilities Work and Asset Management allows all background processes to be restarted or rerun as required. During the execution of the background process, restart information per thread is stored within framework, like a checkpoint. This checkpoint is performed at the last commit point as dictated by the Commit Interval or the Timeout value (Timeout applies to Oracle implementations only). When a commit is performed, the last commit point is recorded for the execution. If a thread of a background process fails, the database automatically rolls back to the last commit point. The thread can then be restarted from that point automatically or from the start of the data. To indicate the restart, the thread is executed with the same parameters as the original.
Additionally, processes are re-runable. A background process is re-runable when you can specify a specific run number to be re-run as required or specify a specific business date to reprocess. Using a rerun number or a previously used business date are all that is required to rerun a process.
Note: Not all background processes use Run number as a run indicator. Refer to the online documentation for which batch processes are re-runable.
Submission Methods
There are several methods for submission supported by Oracle Utilities Work and Asset Management.
• Online Submission - This allows testing users to submit batch jobs from the online component. This facility is typically used for non-production environments.
Note: Oracle strongly recommends that online submission is not used for Production environments.
• Command Line Submission - A number of commands are available to start/stop threadpools and jobs from the command line or from a third party scheduler.
Online Submission
One of the most important useful testing or demonstration facilities of the product is the ability to submit batch processes from the online component of Oracle Utilities Work and Asset Management. An authorized user can submit any batch process using an online batch submission page.
The on-line batch submission page enables you to request a specific background process to be run. Refer to the online help for a discussion on how to use the online submission function. When submitting a background process on-line, you may override standard system parameters and you may be required to supply additional parameters for your specific background process. After submitting your background process, you may use this page to review the status of the submission.
Basically, the following process is used to submit background processes using the online submission method:
• The process to be executed is registered online as to be submitted (or queued). This marks the process execution as Pending. When you request a batch process to be submitted from on-line, the execution of the desired background process will result in the creation of a batch run. Just as with background processes executed through your scheduler, you may use the Batch Run Tree page to view the status of the run, the status of each thread, the run-instances of each thread, and any messages that might have occurred during the run.
Note: Your online submission record is assigned a status value so that you may know whether your batch process has been submitted and whether or not it has ended; however, it will not contain any information about the results of the background process itself. You must navigate to the Batch Run Tree page to view this detail.
• A background process is scheduled using the submission daemon that will pickup any Pending background process executions and execute them. When you save a record on the batch process submission page, the batch process does not get submitted automatically. Rather, it saves a record in the batch process table. A special background process will periodically check this table for pending records and will execute the batch process. This background process will update the status of the batch process submission record so that a user can determine when their batch process is complete.
Note: At installation time, your system administrator will set up this special background process or configure the scheduler daemon to periodically check for pending records in the batch process submission table. Your administrator will define how often the system will look for pending records in this table.
It should be noted that this special background process only submits one pending batch process submission record at a time. It submits a batch process and waits for it to end before submitting the next pending batch process.
Note: If you request a batch process to be run multi-threaded, the special background process will submit the batch process as requested. It will wait for all threads to complete before marking the batch process submission record as ended.
During execution the status of the execution in the Batch Run Tree is updated as well as the original submission screen. If you wish the system to inform you when the background process completes, you may supply your email address. The email you receive will contain details related to the batch process’s output; similar to the batch process results you would see from the Batch Run Tree.
Note: This assumes that during the installation process, your system administrator configured the system to enable email notification. Your administrator may also override the amount of detail included in the email notification.
To use this facility the following must be enabled:
• The online submission daemon and DEFAULT threadpoolworker is configured using the following parameters:
• The BATCHENABLED (see BATCHENABLED - Whether Online Batch Server is Enabled) setting must be set to true.
• The BATCHDAEMON (see BATCHDAEMON - Whether Online Batch Daemon is Enabled) setting must be set to true.
• The BATCHTHREADS (see BATCHTHREADS - Number of Online Batch Server Threads) setting must be greater than zero.
Note: The default setting for BATCHTHREADS is 5 threads. This setting should only be altered if large numbers of testers need to use the DEFAULT threadpoolworker at the same time.
• An external DEFAULT threadpool and daemon is started using the -d Y option on the threadpoolworker (see threadpoolworker.sh - Manage Threadpools) or starttpw.sh (see starttpw.sh - Start Threadpoolworker) command.
For example:
threadpoolworker.sh -p DEFAULT=5 -d Y
Command Line Submission
At a site implementing Oracle Utilities Work and Asset Management, the batch processes to be executed to support the business as well as perform expected maintenance on the system needs to be scheduled, managed and executed from a central point. In most sites, this is done by using a third party batch process scheduler that controls the scheduling and execution of any batch processes across a site.
To support the use of such a scheduler with any Oracle Utilities Application Framework based product, several scripts and related configuration files have been provided to allow the scheduler to execute the batch processes.
The scripts and configuration files allow for three fundamental facilities that can be used by external scheduling tools:
• The interface is command line based (it can also be invoked using a java based API see the product javadocs within AppViewer for a description of the interface) which most external scheduling tools support.
• The command based utilities return a standard return code to indicate the batch process has been successful or has been unsuccessful. Actions dependent on return code within the scheduler can then be configured.
• The logs within the utilities provided are in a common format that can be interrogated by the external scheduler to provide finer grained actions (especially for unsuccessful executions).
For additional advice about interfacing external schedulers with Oracle Utilities Work and Asset Management refer to the
Batch Best Practices (Doc Id:
836362.1) reference paper on
My Oracle Support.
The following guidelines apply:
• Any command should execute splenviron (see splenviron – Set Environment Variables) prior to execution.
• Threadpoolworkers should be started prior to running any batch job threads using the threadpoolworker (see threadpoolworker.sh - Manage Threadpools) or starttpw.sh (see starttpw.sh - Start Threadpoolworker) command in the scheduler.
• Batch Job threads should be submitted using the submitjob (see submitjob.sh - Submit Batch Threads) command.
• If threadpoolworkers need to be shutdown the jmxbatchclient -k (see jmxbatchclient.sh – JMX batch command line) command should be used.