5.1 About the OFSBD Job Protocol
The system initiates all OFSBD jobs by using a standard operational protocol that utilizes each job’s metadata, which resides in a standard set of database tables.
- Dispatcher: Polls the job metadata for new jobs that are ready for execution. This daemon process starts a MANTAS process for each new job.
- Mantas: Creates a new job entry based on a template for the job that has the specific parameters for this execution of the job (that is, it clones a new job).
Table 5-1 OFSBD Job Protocol Shell Scripts
| OFSBD Job Protocol Process Shell Script | Description |
|---|---|
| start_mantas.sh | Starts all OFSBD jobs. This script invokes the cloner and MANTAS processes. This is the integration point for a third-party scheduling tool such as Maestro or AutoSys. |
| start_chkdisp.sh | Calls on the check_dispatch.sh script to ensure that the dispatcher runs. |
| stop_chkdisp.sh | Stops the dispatcher process. |
| restart_mantas.sh | Changes job status codes from the ERR status to the RES status so that the dispatcher can pick up the jobs with the RES status. |
| recover_mantas.sh | Changes job status codes for jobs that were running at the time of a system crash to the ERR status. After running this script, the restart_mantas.sh script must be run to change the ERR status code to RES in order for the dispatcher to be able to pick up these jobs. |
In the OFSBD Job Protocol, the processes use a variety of metadata that the OFSBD database provides. Some of this metadata specifies the jobs and their parameters that are associated with the regular operations of an OFSBD installation. Some of this metadata captures the status of job execution and is useful for monitoring the progress of an OFSBD operational cycle.
Understanding the OFSBD Job Protocol
OFSBD Jobs are created through the Scenario Manager. Jobs are grouped together to run in parallel through Job Template Groups in the KDD_JOB_TEMPLATE table. These templates associate an algorithm to run with parameters that the algorithm requires. Template groups enable you to identify what jobs to run.
Table 5-2 KDD_JOB_TEMPLATE with Sample Job Template Group
| JOB_ID | TEMPLATE_GROUP_ID |
|---|---|
| 37 | 1 |
| 41 | 1 |
Understanding the Dispatcher Process
The dispatcher process polls the job metadata waiting for jobs that must be run. To control system load, the dispatcher also controls the number of jobs that run in parallel.
Generally, the dispatcher process should be running continuously, although it is possible to run jobs without a dispatcher. For each job in the template group, the dispatcher runs a MANTAS process.
The dispatcher tracks jobs for status and completion, and reports any failure to the dispatch log.
Note:
If you observe job failures when running on the AIX operating system, it may be due to resource constraints of the AIX system. In this case, you must try reducing the number of jobs you are attempting to run in parallel or try running the jobs sequentially.Refer to Starting the Dispatcher and Stopping the Dispatcher for more information.
Understanding the MANTAS Process
The dispatcher runs jobs using the MANTAS process. This process runs the appropriate algorithm, tracks status in the KDD_JOB and KDD_RUN tables. One MANTAS process can result in multiple KDD_RUN records.
The MANTAS process also logs job progress and final status.
Applying a Dataset Override
The dataset override feature permits dataset customizations specific to your site, which can be retained outside of the scenario metadata. The override to a dataset definition is stored in a file accessible by the Behavior Detection engine. The dataset override feature allows improved performance tuning and the ability to add filters that are applicable only to your site’s dataset.
When the system runs a job, it retrieves the dataset definition from the database. The Behavior Detection engine looks in the configured directory to locate the defined dataset override. The engine uses the override copy of the dataset instead of the copy stored in the scenario definition in the database, if a dataset override is specified.
- The columns returned by the dataset override must be identical to those returned by the product dataset. Therefore, the dataset override does not support returning different columns for a pattern customization to use.
- The dataset override can use fewer thresholds than the product dataset, but cannot have more thresholds than the product dataset. Only thresholds applied in the dataset from the scenario are applied.