Batch Report Generation Process

Parallel processing has been introduced into the Batch Report Generation AG Process to enable its improved performance.

This has been made possible by introducing a new AG process called BATCH REPORT GENERATION WORKER.

The Batch Generation Argus Safety Service process has been split into two separate processes:

1. BATCH REPORT GENERATION
2. BATCH REPORT GENERATION WORKER

Both the above processes need to be configured to generate Batch Reports.

Note:

The configuration for BATCH REPORT GENERATION remains the same as is.

In addition to the above process, you need to now configure the new BATCH REPORT GENERATION WORKER process as well.

Multiple instances of the new BATCH REPORT GENERATION WORKER process must be configured either on the same AG Service box, or on multiple Argus Safety Service boxes, to achieve better performance.

If multiple instances are configured, each process of type "BATCH REPORT GENERATION WORKER" should be configured with a different Argus Safety Service user.

Details

The BATCH REPORT GENERATION process will populate the reports (Expedited Reports only) that need to be processed by the BATCH REPORT GENERATION WORKER in a new table BATCH_GENERATION_QUEUE.

The Report IDs from CMN_REG_REPORTS are used to populate this new table. The status column in this table indicates to the batch report generation and worker processes about how to process the corresponding reports.

It will have one of the following values:

Value: Status	Denotes that...
0: Pending	The record has been populated by the Batch Report Generation process but has not yet been picked up by the Worker process.
1: Processing	The record has been picked up by a worker process for generation.
2: Completed	The report has been successfully generated.
3: Failed	The report generation failed.

If the report generation fails, another record for the same Report ID is inserted in the queue table during the next run of the BATCH REPORT GENERATION process so that the Report generation is tried again.

At any point of time, there will not be more than two records in the queue table with the same REG_REPORT_ID.

If a record in the queue table remains in Status "1" for more than 2 hours, then the Status is marked as "3" during the next run of BATCH REPORT GENERATION process.

The records with Status "2" get removed in the next run of the BATCH REPORT GENERATION process.

The records with Status "3" get retried once by the process.

After that, the report is not retried for 8 hours and will get removed after the 8 hour window.

Note that the failed records are kept for 8 hours in order to prevent the same record, which may have failed due to configuration or data issues, being retried continuously.

Once the failed records are cleared after 8 hours, the report will be retried for generation. The 8 hour cycle will repeat indefinitely until the report generates successfully or until the report is deleted.

For expedited reports that are set to generate on DLP, when the BATCH REPORT GENERATION process runs and sees that there are still DLP Reports yet to be picked by a worker process or is still being generated, it will exit without adding any more DLP reports to the queue.

Once all the DLP reports (RUN_ON_DLP = 1) are processed (i.e., have STATUS = 2 or 3), the BATCH REPORT GENERATION process will insert more DLP reports to the queue table.

Note:

The logs of Success/Failed count for the Batch Generation process are now distributed across multiple processes. Getting a cumulative count is not possible. Customers need to check the BATCH_GENERATION_QUEUE table to see if there are failed records which have failed consistently.

The following table lists the structure of the queue table:

Column Name	Data Type	Nullable	Comments
ID	NUMBER	No	This is the Primary Key of the table, and the values come from the sequence S_BATCH_GENERATION_QUEUE.
BATCH_ID	NUMBER	No	This column holds the value of BATCH_ID which comes from CFG_REPORT_SCHEDULE.ID
PROCESSING_USER_ID	NUMBER	No	This column holds the value of the AG Service User ID of the process which picks up this record for processing.
STATUS	NUMBER	Yes	This column can hold the following values: 0:Pending, 1: Processing, 2: Completed, 3: Failed.
REG_REPORT_ID	NUMBER	No	The data for this column comes from CMN_REG_REPORTS.REG_REPORT_ID
REPORT_FORM_ID	NUMBER	Yes	The data for this column comes from CMN_REG_REPORTS.REPORT_FORM_ID
SESSION_USER_ID	NUMBER	Yes	For Non-DLP Reports, the value for this column is the AG Service User ID. For DLP Reports, it is the Lot User ID.
STATE_ID	NUMBER	Yes	The data for this column comes from CMN_REG_REPORTS.STATE_ID
LAST_PROCESSING_TIME	DATE	Yes	The time at which the record was last picked up for processing.
RUN_ON_DLP	NUMBER	Yes	The value of 1 means the report generation should run on DLP. If not, the value should be 0.
ERROR_TEXT	VARCHAR2 (4000 CHAR)	Yes	The worker process generates the reports and if it fails, it updates the error text (if any). The worker process does not know whether the report generation has been successful. Hence, it sets the error_text as "Success value not available", and leaves the record there itself.

Configuration

Some of the parameters described above (such as the 8 hour cycle, the number of times the failed reports get retried, etc.) can be controlled through settings in agservice.ini.

More information can be found in the Default values that have been listed below:

[Batch Generation Configuration]

Max Retry Attempts = 1

This is the number of times that a given reg_report_id gets added in the queue in a given 8 hour slot. It gets added only if the first attempt results in a failure (status = 3).

The above setting indicates that if a report fails during generation, another record for the same report id is added to the queue table for the same reg_report_id, in spite of the failed record still being there in the queue (with status = 3).

This enables the report generation to be retried. If the second attempt also fails, then that report is not added to the queue again, until the queue is cleared (which happens once in 8 hours).

[Batch Generation Configuration]

Max Failed After Minutes = 120

This value is used to identify crashed reports. If a report in the queue is still in status 1 (which means "in processing") even after so many minutes, it means that the EXE would have crashed.

Configure this value to the maximum number of minutes that a report generation would take in your environment, plus 1.

[Batch Generation Configuration]

Clear Failed Records After Hours = 8

Failed reports are not inserted multiple times (for more than the configured number of retries) as long as they remain in the queue table (with status 3).

However, these records are automatically deleted after 8 hours. This is done to enable retrying the failed records again.

In the duration of 8 hours, there is a possibility that users would have corrected the case data and the reports might succeed now.

Set this value to a realistic number, as applicable.

If you set the value to an unrealistic number, in case of any report failures because of data problem, the Argus Safety Service Process continues attempting the generation of the same report again and again, thus blocking other newer reports from getting generated.

[Batch Generation Configuration]

Max Iterations = 10

This is the number of reports that a worker process tries to generate in a single run. When the worker process gets instantiated, it locks one record at a time from the queue table, and starts processing it.

Some of the reports may fail, while the others may succeed. After processing the configured number of reports (default 10), the process exits.

The next instance of the worker process comes up after the time interval that has been configured in the AG Config utility.

Configure the above value to a higher number if the process has enough speed to generate more number of reports in the time duration before the next instance of the process starts.

You must also ensure that you verify when the agproc.exe is using maximum memory and becoming slow.

If you find that the EXE would consume a lot of memory and might slow down by the time it generates 20 reports, then configure the number to something smaller.

Whatever the configured number is, note down how much time it is taking to process so many records, and configure the frequency of that process to start another instance soon, after those many seconds/minutes.