19.19 Event Purge

This section describes the Event Purge.

Sometimes, events are ingested multiple times or generated by poor data quality during the processing.In these instances, you can clear the ingested data for better correlations and investigations. This is applicable to both cases and pre-cases. When an event purge is performed for the case, the linked events and entities will be deleted. If an entity is linked to both valid and purged events, then the entity will not be removed.

There are two types of the purge:

  • Individual Event Purge: Individual events that are identified with bad data. Providen_event_skey generated in the consolidation layer and relevant comments to purge the event.
  • Batch Purge: Enter the purge type as “BATCH” in the v_purge_type column and run skey in n_run_skey column of FCC_PURGE_INPUT tableif you want to purge a complete batch. When a batch is purged, the events linked to that batch will be purged.

Below is the list of features:

  • Supports purging individual and batch events.
  • Removing purged events and related entities from the case
  • Capturing the audit log of the purged events related to the case.
  • Purging case (in Extendible status) if all the events in the case are purged.
    • Events Purged before promoting to the case: If the event is part of an un-promoted correlation, then drop the event and associated entities from the correlation. The events which are purged will not take part in the future correlation process. If the events in correlation after purging are disconnected, then re-run correlation for these events.
    • Events Purged after promoting to Case: If any event is part of the case, then drop the event and associated business entities from the case. If the associated business entities (for example, customer)of the purged event(s) are associated with other events in the case, then those will remain in the business tab. By default, only events that are part of the case in Extendible status can be purged. The events which are purged will not take part in the future correlation process.

Event Purging Using Tables

Perform the following steps to purge events using table updates:

  1. Configure the parameters in table FCC_PURGE_INPUT. In this table, data with the event skey column can be added. For this, follow the below steps:
  2. Locate EventPurge.cfg ( path: <installed area>/ficdb/conf). Modify thefollowing details:
    • Status: Status of case
    • Batchsize: Defines the maximum size of events in a batch
    • TestMode: If the testmode is True, then data will not be purged but purging is done for testing purpose.
  3. Look at the existing table FCC_EVENTS. In table FCC_PURGE_INPUT, the event skey should be the same as mentioned in table FCC_EVENTS.

    Table 19-15 FCC_PURGE_INPUT Table

    Column Name Primary Key Column Type Nullable
    n_err_seq_id NUMBER No
    n_event_skey NUMBER(22)
    n_run_skey NUMBER(22)
    v_user_comments VARCHAR2(4000 CHAR)
    f_purge_success_flag CHAR(1 CHAR)
    d_requested_date DATE
    d_fic_mis_date DATE
    v_user_id VARCHAR2(50 CHAR)
    v_data_origin VARCHAR2(30 CHAR)
    d_prcsng_batch_date DATE
    v_purge_type VARCHAR2(20 CHAR)
    • n_err_seq_id: Enter the sequence ID. This field accepts only numeric values. This is a mandatory field.
    • n_event_skey: Enter the skey of the event which you want to purge. This is a mandatory field if it is an individual event purge.
    • n_run_skey: Enter the run skey of the batch you want to purge. This is a mandatory field if it is a complete batch event purge.
    • v_user_comments: User comments if required
    • f_purge_success_flag: Defines the purge success flag. If the purge is failed, then this flagwill be displayed as E. By default, it has to be set to Null or N. This flag turns to Y if the purge is successful. If purge fails (flag = E), then check and correct it, and update this entry as Null or N and again execute the purge.
    • d_requested_date: Date of individual event purge request
    • d_fic_mis_date: Date of BATCH. This is applicable only if n_run_skey is defined.
    • v_user_id: User ID who is performing the purge.
    • v_data_origin: Define the data region, like US, IND, and so on. This is applicable only for thebatch purge. For example, a batch can have data for multiple data origins and if you want to delete data only for the India region, then define the region in this field.
    • d_prcsng_batch_date: Date on which the purge process request is completed.
    • v_purge_type: Define the type of purge. It can be a BATCH or EVENT.
  4. Execute the script: EventPurge.sh

Hint Configuration in Event Purge (EventPurgeConfig.cfg)

The EventPurgeConfig.cfg file is available in the #FIC_HOME#/ficdb/conf folder. This contains the list of tables that has to be purged in the following format:

ActualTableName:PurgeTableName:deleteType:deleteKey

Eg: KDD_CASES:KDD_CASES_PE:directDelete:case_intrl_id

The table-wise Hint and Global Hint Configurations are introduced to improve Insert/Delete Queries. If table-wise configuration is available, it will be used; else, Global Hint configuration will be used.

  • Table wise Hint Configuration
    • To optimize the Insert and Delete queries, the above format will additionally support the optimizer hint for insert and delete operations separated by semicolon (:).You can configure the required hint as per your optimizer:

      ActualTableName:PurgeTableName:deleteType:deleteKey:optimizerHintInsert:optimizerHintDelete

      E.g.: KDD_CASES:KDD_CASES_PE:directDelete:case_intrl_id:/*+ parallel(8) */:/*+ parallel(8) */

    • To optimize only for insert scripts, use the below format:

      ActualTableName:PurgeTableName:deleteType:deleteKey:optimizerHintInsert

      E.g.: KDD_CASES:KDD_CASES_PE:directDelete:case_intrl_id:/*+ parallel(8) */

    • To optimize only for delete scripts, use the below format:

      ActualTableName:PurgeTableName:deleteType:deleteKey::optimizerHintDelete

      E.g.: KDD_CASES:KDD_CASES_PE:directDelete:case_intrl_id::/*+ parallel(8) */

  • Global Hint Configuration

    The following configurations must be used to apply any global optimizer hint. You can set up the required hint as per your optimizer.

    • For Delete operation:

      OptimizerHintDelGlobal: - To apply the optimizer hint to all the tables provided in cfg files for delete scripts.

      E.g. OptimizerHintDelGlobal:/*+ parallel(4) */

    • For Insert operation:

      OptimizerHintInsGlobal:- To apply the optimizer hint to all the table’s information provided in cfg files for insert scripts

      E.g. OptimizerHintInsGlobal:/*+ parallel(4) */