6.6 Historical Data Copy

TBAML maintains records that are directly involved with detected behaviors in a set of archive, or ARC, tables. The Historical Data Copy (HDC) process identifies the records against which the current batch’s scenario runs generated events and copies them to the ARC tables.

The run_hdc.ksh and upd_kdd_review_fin.sh must run upon completion of all detection and other event post-processing , such as scoring and assignment, but before the system ends the batch with the following shell script:

end_mantas_batch.sh

This script is part of the Database Tools that reside in the <OFSAAI Installed Directory>/ database/db_tools/bin directory.

The run_hdc.ksh shell script manages the HDC process. This process connects to the database as the user that the truncate.database.username property identifies in the <OFSAAI Installed Directory>/database/db_tools/mantas_cfg/install.cfg file. This property should identify the Atomic Schema user, a user in the database with write access to tables in Behavior detection Atomic schema.

To improve performance, you can adjust two configurable parameters in the <OFSAAI Installed Directory>/database/db_tools/mantas_cfg/install.cfg file.

Table 6-1 HDC Configurable Parameters

Parameter	Recommended Value	Descriptions
hdc.batchsize	10000	Number of break match key IDs are included in each batch thread for data retrieval.
hdc.maxthreads	2x (Number of CPUs)	Maximum number of concurrent threads that HDC uses for retrieving data to tune performance.

To run the Historical Data Copy (HDC) process, follow these steps:

1. Navigate to <OFSAA installed directory>/database/db_tools/bin/execute run_hdcTBAML.ksh.

   By default, log messages for this script are written in the <OFSAAI Installed Directory>/ database/db_tools/logs/hdc.log file.

2. Verify TBAML-related tables to check the HDC data copy.