D.2 Flat File Ingestion
This section refers to Behavior Detection () Ingestion Flat Files and covers the
following topics:
BDF.xml File Parameters
The following table describes the parameters which must be configured in the BDF.xml
file under the <OFSAAI Installed Directory>/bdf/config folder
for processing DIS files.
Table D-3 Parameters Related to Processing DIS Files
| Property Name | Description | Default |
|---|---|---|
| DIS.Source |
Indicates the source of DIS records. Valid values are:
|
FILE |
| DIS.ArchiveFlag | Indicates whether a DIS file should be archived after it has been processed. | true |
| DIS.BufferSize | Indicates the size of a byte buffer (in kilobytes) used to read in a line from a DIS file. This should be set to the maximum possible record size (in kilobytes) of a record in a DIS file. | 100 |
| DIS.InputFileCharset | Indicates the character set of a DIS file. | UTF8 |
| DIS.Default.Check.Requirement | Indicates whether the mandatory and conditional checks on a DIS record should be done | true |
| DIS.Default.Reject.Requirement | Indicates whether a mandatory or conditional check failure for a record should result in the record being rejected. If this is set to FALSE and a missing value is attempted to be inserted into a NOT NULL column, then the record will be rejected anyway. | true |
| DIS.Default.Check.Domain | Indicates whether the domain value checks on a DIS record should be done. | true |
| DIS.Default.Reject.Domain | Indicates whether a domain value check failure for a record should result in the record being rejected. | true |
| DIS.Default.Check.Length | Indicates whether the maximum length checks on a DIS record should be done. | true |
| DIS.Default.Reject.Length | Indicates whether a maximum length check failure for a record should result in the record being rejected. If this is set to FALSE, then the value will be truncated based on the maximum length of the field. | true |
| DIS.Default.Check.Threshold | Indicates whether the threshold checks (GREATER_THAN_ZERO,etc) on a DIS record should be done. | true |
| DIS.Default.Reject.Threshold | Indicates whether a threshold check failure for a record should result in the record being rejected. | true |
| DIS.Default.Check.Lookup | Indicates whether the reference data lookups on a DIS record should be done. | true |
| DIS.Default.Reject.Lookup | Indicates whether a reference data lookup failure for a record should result in the record being rejected. | true |
| MITrxnProducttypes | Indicates the parameter which is used to pass a list of product codes for trailing digit purpose (AUG_INSTR_NB derivation). |
|
| CustProfileLookBack |
Indicates the parameter which is used to look back at the days in Customer Summary Daily for Customer Summary Month recalculation. Note:Inorder to look back at a specific time period in Customer Summary Daily, you must have partitions available in Customer Summary Month. |
31 |
| CustAcctHolder Type | Indicates the parameter which is used to identify customer account types to be included in customer summary. | CI |
Ingest DIS Data Files by Group
Ingestion Manager processes data files in groups (in a specified order) from Oracle
client data in the / inbox directory. The following list of files can be run using
CSA. Files have been grouped in such a way that files in the same group can be
executed in parallel to load data. However, you must execute Group 1 through Group 6
in sequence.
Table D-4 Ingest DIS Data Files By Group
| Group | Data Files | - |
|---|---|---|
| 16 |
.Account Phone Watch List Account Email Address Front Office Transaction |
Account Customer Role Organization Country |
| 17 | .Account Customer |
WatchList Entry Front Office Transaction Party |
| 18 |
.Account To Customer Account Balance Account Address Customer To Customer Relationship Customer Phone |
Customer Email Address Customer Country Customer Address Controlling Customer |
| 19 | Back Office Transaction | |
| 20 | Open Order
Order |
TradeExecutionEvent |
TBAML Flat File Interface
The following tables describe the Ingestion Flat File details for products within the TBAML Application Pack. Files have been grouped in such a way that files in the same group can be executed in parallel to load data. However, you must execute Group 1 through Group 5 in sequence. For more information, see List of Data Quality Group, T2T and H2T Names.
The Staging Representation column indicates whether this file requires a Staging source.
Table D-5 Group 1 Interface Ingestion Flat Files
| Interface File Name | Current Ingestion | Staging Representation | DQ | T2T |
|---|---|---|---|---|
| Account Phone | Datamaps | Yes | - | Yes |
| Account Email Address | Datamaps | Yes | - | Yes |
| Insurance Policy | Datamaps | Yes | - | Yes |
| Insurance Policy Balance | Datamaps | Yes | - | Yes |
| Account Customer Role | Datamaps | Yes | - | Yes |
| Insurance Policy Feature | Datamaps | Yes | - | Yes |
| Insurance Policy to Customer | Datamaps | Yes | - | Yes |
| Loan | Datamaps | Yes | - | Yes |
| Loan Daily Activity | Datamaps | Yes | - | Yes |
| Online Account | Datamaps | Yes | - | Yes |
| Insurance Seller | Datamaps | Yes | - | Yes |
| Insurance Seller to License | Datamaps | Yes | - | Yes |
| Country | Datamaps | Yes | - | Yes |
| WatchList | Datamaps | Yes | - | Yes |
| Insurance Product | Datamaps | Yes | - | Yes |
| Insurance Transaction | Datamaps | Yes | - | Yes |
| Front Office Transaction | Datamaps | Yes | - | Yes |
| Organization | Datamaps | Yes | - | No |
| Market Center | Datamaps | Yes | - | No |
| Market Index Daily | Datamaps | Yes | - | No |
| Issuer | Datamaps | Yes | - | No |
| Market Index | Datamaps | Yes | - | No |
| Service Team Member | Datamaps | Yes | - | No |
| Service Team | Datamaps | Yes | - | No |
| CTR Transaction | runDP/runDL | No | - | No |
| Account Realized Profit and Loss | runDP/runDL | No | - | No |
| Letter of Intent | runDP/runDL | No | - | No |
| Collateral Value-Currency | runDP/runDL | No | - | No |
| Collateral Value-Product | runDP/runDL | No | - | No |
| Commission Product | runDP/runDL | No | - | No |
| Compliant Registration | runDP/runDL | No | - | No |
| Complaint Type Rating | runDP/runDL | No | - | No |
| Employee to Insurance Policy | runDP/runDL | No | - | No |
| Investment Guideline | runDP/runDL | No | - | No |
| Investment Guideline to Account | runDP/runDL | No | - | No |
| System Logon Type | runDP/runDL | No | - | No |
| Registered Representative Complaint | runDP/runDL | No | - | No |
| Energy And Commodity Instrument | runDP/runDL | No | - | No |
Table D-6 Group2 Interface Ingestion Flat Files
| Interface File Name | Current Ingestion | Staging Representation | DQ | T2T |
|---|---|---|---|---|
| Account to Peer Group | Datamaps | Yes | - | Yes |
| Account Group | Datamaps | Yes | - | Yes |
| Peer Group | Datamaps | Yes | - | Yes |
| Security Market Daily | Datamaps | Yes | - | No |
| Security Firm Daily | Datamaps | Yes | - | No |
| Security | Datamaps | Yes | - | No |
| Market Index Member Security | Datamaps | Yes | - | No |
| Security Market State Change | Datamaps | Yes | - | No |
| Matched Entity | runDP/runDL | No | - | No |
| Trusted Pair | Datamaps | Yes | - | No |
| Firm Account Position Pair | runDP/runDL | No | - | No |
| Natural Gas Flow | runDP/runDL | No | - | No |
Table D-7 Group 3 Interface Ingestion Flat Files
| Interface File Name | Current Ingestion | Staging Representation | DQ | T2T |
|---|---|---|---|---|
| Account | Datamaps | Yes | - | Yes |
| Customer | Datamaps | Yes | - | Yes |
| WatchList Entry | Datamaps | Yes | - | Yes |
| Loan Product | Datamaps | Yes | - | Yes |
| Employee | Datamaps | Yes | - | Yes |
| Front Office Transaction Party | Datamaps | Yes | - | Yes |
| Organization Relationship | Datamaps | Yes | - | No |
| Restriction List | Datamaps | Yes | - | No |
| Automated Quote | Datamaps | No | - | No |
| Account Supplemental Attribute | runDP/runDL | No | - | No |
| Customer Supplemental Attribute | runDP/runDL | No | - | Yes |
| Market Trading Session | runDP/runDL | No | - | No |
| Account Group Address | runDP/runDL | No | - | No |
| Account Group Investment Objective | runDP/runDL | No | - | No |
| Account Group IOS Member | runDP/runDL | No | - | No |
| Account Group Member Experience | runDP/runDL | No | - | No |
| Loan Origination Action | runDP/runDL | No | - | No |
| Mail Handling Instruction Activity | runDP/runDL | No | - | No |
| Banker To Officer | runDP/runDL | No | - | No |
| Reference Table Detail | runDP/runDL | No | - | No |
| General Usage List | runDP/runDL | No | - | No |
| Loan Origination Product | runDP/runDL | No | - | No |
| Organization To Mortgage Type | runDP/runDL | No | - | No |
| Securities License | runDP/runDL | No | - | No |
| Service Vendor | runDP/runDL | No | - | No |
| Energy and Commodity Trade | runDP/runDL | No | - | No |
Table D-8 Group4 Interface Ingestion Flat Files
| Interface File Name | Current Ingestion | Staging Representation | DQ | T2T |
|---|---|---|---|---|
| Market News Event | Datamaps | No | - | No |
| Managed Account | Datamaps | Yes | - | No |
| Account To Customer | Datamaps | Yes | - | Yes |
| Branch CTR Transaction | Datamaps | Yes | - | No |
| Branch CTR Conductor | Datamaps | Yes | - | No |
| Branch CTR Summary | Datamaps | Yes | - | No |
| Account Group Member | Datamaps | Yes | - | Yes |
| Account To Correspondent | Datamaps | Yes | - | Yes |
| Account Balance | Datamaps | Yes | - | Yes |
| Account Address | Datamaps | Yes | - | Yes |
| Customer Identification Document | Datamaps | Yes | - | Yes |
| Customer To Markets Served | Datamaps | Yes | - | Yes |
| Customer To Products Offered | Datamaps | Yes | - | Yes |
| Customer To Customer Relationship | Datamaps | Yes | - | Yes |
| Anticipatory Profile | Datamaps | Yes | - | Yes |
| Customer Phone | Datamaps | Yes | - | Yes |
| Customer Email Address | Datamaps | Yes | - | Yes |
| Customer Country | Datamaps | Yes | - | Yes |
| Customer Address | Datamaps | Yes | - | Yes |
| Online Account to Account | Datamaps | Yes | - | No |
| Controlling Customer | Datamaps | Yes | - | No |
| Employee To Account | Datamaps | Yes | - | Yes |
| Account Position | Datamaps | Yes | - | No |
| Security Trading Restriction | Datamaps | Yes | - | No |
| Employee Trading Restriction | Datamaps | Yes | - | No |
| Employee Phone | Datamaps | Yes | - | Yes |
| Employee Email Address | Datamaps | Yes | - | Yes |
| Employee Address | Datamaps | Yes | - | Yes |
| Outside Business Activity | Datamaps | Yes | - | No |
| Private Security Transaction | Datamaps | Yes | - | No |
| Security Group Member | Datamaps | Yes | - | No |
| Security Investment Rating | Datamaps | Yes | - | No |
| Structured Deal | Datamaps | Yes | - | No |
| Account Profit and Loss | Datamaps | Yes | - | No |
| Account Position Pair | Datamaps | Yes | - | No |
| Account Investment Objective | Datamaps | Yes | - | No |
| Mutual Fund Break point | Datamaps | Yes | - | No |
| Account Feature | runDP/runDL | No | - | No |
| Access Events | runDP/runDL | No | - | No |
| Customer Balance | runDP/runDL | No | - | No |
| Front Office Transaction Remittance Document | runDP/runDL | No | - | No |
| Related Front Office Transaction Information | runDP/runDL | No | - | No |
| Account To Organization | runDP/runDL | No | - | No |
| Firm Account Position | runDP/runDL | No | - | No |
| External Investment Account Position | runDP/runDL | No | - | No |
| Employee To Organization | runDP/runDL | No | - | No |
| Security Select List Entry | runDP/runDL | No | - | No |
| Account Fees | runDP/runDL | No | - | No |
| Account Profile Stage | runDP/runDL | No | - | No |
| Account Qualification Agreement | runDP/runDL | No | - | No |
| Account Representative Position | runDP/runDL | No | - | No |
| Account Asset Allocation | runDP/runDL | No | - | No |
| Account Scheduled Event | runDP/runDL | No | - | No |
| Account Identifier Change History | runDP/runDL | No | - | No |
| Account Position Profile And Loss | runDP/runDL | No | - | No |
| Uncovered Option Account Position | runDP/runDL | No | - | No |
| Account Collateral | runDP/runDL | No | - | No |
| Mail Handling Instruction | runDP/runDL | No | - | No |
| Mutual Fund Family Letter of Intent | runDP/runDL | No | - | No |
| Employee Disciplinary Action | runDP/runDL | No | - | No |
| Employee Exam History | runDP/runDL | No | - | No |
| Employee Firm Transfer History | runDP/runDL | No | - | No |
| Employee Securities License State Registration | runDP/runDL | No | - | No |
| Employee Supervision List | runDP/runDL | No | - | No |
| Employee To Manager History | runDP/runDL | No | - | No |
| Employee To Securities License | runDP/runDL | No | - | No |
| Employment History | runDP/runDL | No | - | No |
| System Logon | runDP/runDL | No | - | No |
| Plan of Solicitation | runDP/runDL | No | - | No |
| Mutual Fund Family Configuration | runDP/runDL | No | - | No |
| Energy And Commodity Market Daily | runDP/runDL | No | - | No |
| Energy And Commodity Firm Daily | runDP/runDL | No | - | No |
| Energy And Commodity Reported Market Sale | runDP/runDL | No | - | No |
| Energy And Commodity Market Trading Session | runDP/runDL | No | - | No |
| Energy And Commodity Market Center | runDP/runDL | No | - | No |
| Energy And Commodity Location | runDP/runDL | No | - | No |
| Energy Flow Mode | runDP/runDL | No | - | No |
| Energy and Commodity Instrument Position | runDP/runDL | No | - | No |
Table D-9 Group 5 Interface Ingestion Flat Files
| Interface File Name | Current Ingestion | Staging Representation | DQ | T2T |
|---|---|---|---|---|
| Borrower | Datamaps | Yes | - | No |
| Back Office Transaction | Datamaps | Yes | - | Yes |
| Account Restriction | Datamaps | Yes | - | No |
| Investment Advisor | Datamaps | Yes | - | No |
| Investment Guideline Override | Datamaps | Yes | - | No |
| Settlement Instruction | Datamaps | Yes | - | No |
| Loan Origination Document Print Log | Datamaps | Yes | - | No |
| Change Log | runDP/runDL | No | - | No |
| Options Violation | runDP/runDL | No | - | No |
| Loan Origination Condition | runDP/runDL | No | - | No |
| Loan Origination Fee Detail | runDP/runDL | No | - | No |
| Loan Origination Note | runDP/runDL | No | - | No |
| Loan Origination To Service | runDP/runDL | No | - | No |
| Investment Guideline Override | runDP/runDL | No | - | No |
| Loan Origination Condition Type | runDP/runDL | No | - | No |
| System Logon To System Logon Type | runDP/runDL | No | - | No |
| System Logon To Organization | runDP/runDL | No | - | No |
| Registered Representative Account Commission | runDP/runDL | No | - | No |
| Registered Representative Account Commission Prior Year | runDP/runDL | No | - | No |
| Registered Representative Commission Monthly Profile | runDP/runDL | No | - | No |
| Registered Representative Commission Product | runDP/runDL | No | - | No |
| Currency Transaction | Datamaps | Yes | - | No |
Table D-10 Group 6 Interface Ingestion for Market Data
| Interface File Name | Current Ingestion | Staging Representation | DQ | T2T |
|---|---|---|---|---|
| Inside Quote | Datamaps | Yes | - | No |
| Market Center Quote | Datamaps | Yes | - | No |
| Reported Market Sale | Datamaps | Yes | - | No |
| Inside Quote_Derived | Datamaps | Yes | - | No |
| Market Center Quote_Deriv ed | Datamaps | Yes | - | No |
| Reported MarketSale_Deriv ed | Datamaps | Yes | - | No |
Table D-11 Group 7 Interface Ingestion for Trade Finance Data
| Interface File Name | Current Ingestion | Staging Representation | DQ | T2T |
|---|---|---|---|---|
| TradeFinanceContractEventAcknowledgementStage | Datamaps | Yes | - | No |
| TradeFinanceContractAmendmentStatusStage | Datamaps | Yes | - | No |
| TradeFinanceContract | Datamaps | Yes | - | No |
| TradeFinancetoAccount | Datamaps | Yes | - | No |
| TradeFinanceDocument | Datamaps | Yes | - | No |
| TradeFinanceGoodorService | Datamaps | Yes | - | No |
| TradeFinanceParty | Datamaps | Yes | - | No |
| DocCollectionContractAcknowlegementStage | Datamaps | Yes | - | No |
| DocumentaryCollectionContractAcceptanceStage | Datamaps | Yes | - | No |
| DocumentaryCollectionDiscrepancyDetail | Datamaps | Yes | - | No |
| DocumentaryCollectionContractEvent | Datamaps | Yes | - | No |