4.4.1 Generating Change Logs with BD
Change log and Change log summary records with BD will be generated through BD.
When loading referential DIS files that are defined as Overwrite, it is possible for BD to generate Change Log records which signify when certain fields associated with a reference data entity have changed. This is done by comparing the contents of the DIS file with the current contents of the associated database table. For performance reasons, this change log processing can be done when external tables are used to load the DIS files, so it is a requirement that DIS.Source=FILE-EXT. This requires an external directory, which is created during installation. In order to give access to an Oracle user, place the .dat files in the external directory.
Note:
To derive the change log records the change log parameters in<OFSAAI Installed Directory>/BDF/config/BDF.xml should be
uncommented. The change log can only be derived from the second day onwards. Since
change log functionality derives changes by comparing the data of two days, the
first day data acts as a reference against which the second day data is compared and
changes are derived.
- Compare fields on a single reference data record that can be identified by a
primary key.
For example, an Account record can be identified by an Account Identifier. When an Account file is ingested, the Primary Customer Identifier on Account XYZ is compared to the Primary Customer Identifier currently in the database for Account XYZ. If they are different, then a Change Log record is created. This process only accounts for updates to already existing records. Change Log records are not created for new reference data records or deleted reference data records.
- Compare the set of values for a given field on several reference data records
that map to a given key.
For example, an Account Address record is identified with a combination of Account Identifier and Address Record Number. However, the information required is whether an Account Address record for a given Account has a field value that is different than any other Account Address record for that Account. For example, every Account Address record has a Country field. If there are two Account Address records for Account XYZ in the database with values for Country of US and CN, respectively. On the next day, an Account Address file is processed and there is an Account Address for Account XYZ with a value for Country of IR. A Change Log record is generated for the Country field of this Account Address record. Furthermore, in the case of Account Address, it is not just the Account Identifier of an Account Address record that is of interest. The Address Purpose is also of interest. So when we look in the database for Account Address records that match a given Account Address record in a DIS file, we look to match both the Account Identifier field and the Address Purpose field.
This processing is controlled by parameters in <OFSAAI Installed
Directory>/bdf/config/BDF.xml. All of these parameters have been commented
out, which means change log processing is turned off by default. To derive the change
log records if DIS.Source = 'FILE-EXT', the relevant parameters
for the DIS files of interest should be copied to <OFSAAI Installed
Directory>/bdf/config/custom/BDF.xml and uncommented.
Table 4-1 Change Log Parameters
| Parameter | Description |
|---|---|
| ChangeLog.<DIS File Type>.Fields | The fields of this particular DIS file type which will be monitored for changes. |
| ChangeLog.<DIS File Type>.IsSet | Whether change log records are generated based on mechanism 1 above (false) or mechanism 2 (true). The default is false. |
| ChangeLog.<DIS File Type>.QueryKey |
This is only relevant when IsSet=true. This defines the key that is used to query for reference data records matching the given one. In the Account Address example given above, the value would be AccountIdentifier,AddressPurpose. If this parameter is not present, then the business key located in the given DIS file type's data map (for example bdf/datamaps/ AccountAddress.xml) is used. |
| ChangeLog.<DIS File Type>.OutputKey | This is only relevant when IsSet=true. This defines the set of fields that are mappedto the Key1, Key2, Key3, and Key4 fields of a Change Log record. This can be different from the QueryKey and business key in order to match what is expected in Change Log DIS file records, and also to support the Change Log Summary data maps. If this parameter is not present, then the business key located in the given DIS file type's data map (for example, bdf/datamaps/ AccountAddress.xml) is used. |
To turn on Change Log processing for a given DIS file type, all the parameters for that file type must be uncommented. The values of the ChangeLog.<DIS File Type>.Fields parameter are preset based on the needs of the KYC application. If different fields are required, then this parameter should be changed. It is not necessary to change any of the other parameters.
For Example: If Address Street line fields are to be considered for change log generation, then the ChangeLog.<DIS File Type>. Fields parameter should be changed for that particular table as shown below.
<Parameter name="ChangeLog.AccountAddress.Fields" type ="STRING"
value="Country,Region,State,City,PostalCode,MailHandlingInstruction"
list="true"/><Parameter name="ChangeLog.AccountAddress.Fields"
type ="STRING" value="Country,Region,State,City,PostalCode,MailHandlingInstruction,
StreetLine1,StreetLine2,StreetLine3,StreetLine4,StreetLine5,StreetLine6"
list="true"/>As in the example above, StreetLine1,StreetLine2,StreetLine3,StreetLine4,StreetLine5 and StreetLine6 will also be considered for change log generation. Similar steps can be followed for other change log related tables well.
Change Log records are written to the CHG_LOG table as the DIS file is being loaded. There are no additional scripts to be run. As soon as the parameters are uncommented, Change Log records are generated the next time DIS files are loaded.