D.3 Directory Structure

The Datamap component is organized as subdirectories below the <OFSAAI Installed Directory>/ bdf file.

Table D-12 Directory Structure Description

Directory Name Description
scripts Shell scripts for running components, setting the environment, and changing passwords
logs Log files containing status and error messages produced by components
config Files used to configure components
config/datamaps XML files containing data map definitions for individual components
jars Java Archive (JAR) files used to run components
data/errors Files containing error records produced by components
data/temp Temporary files produced by components
inbox Data files provided by the Oracle client in DIS format
fuzzy_match C++library files used for the purpose of fuzzy matching names

Figure D-1 Subsystem Directory Structure


Description of Figure D-1 follows
Description of "Figure D-1 Subsystem Directory Structure"

The following sections describe the directory structure.

Scripts

The scripts folder contains the following files:
  • changePassword.sh - Changes passwords used during the execution of components. Refer to the Installation Guide for more information.
  • env.sh - Sets ups the shell environment of components.
  • execute.sh - Executes components.
For Example:
<OFSAAI Installed Directory>/bdf/scripts/execute.sh <component>
<OFSAAI Installed Directory>/bdf/scripts/execute.sh CorrespondentBankProfile

Note:

Component in this document means a batch process which is part of the Datamap subsystem. For the most part, these components will refer to XML data maps. For example, the AccountProfile_Balance component refers to the AccountProfile_Balance.xml data map.

Running these files in the subsystem improves performance time.

Logs

The log file has information about the warnings, errors, and status of the component. Additional information can be obtained from a component by turning on diagnostic logging. This can be done by setting the Log.DIAGNOSTIC.Enabled parameter to true. In a production environment, this should be left as false and only changed to true when debugging errors or performance issues.

Log files for each component are written to a log file named for the component inside a subdirectory of the logs directory named for the current processing date in YYYYMMDD format:

For example:

<OFSAAI Installed Directory>/bdf/logs/<processing date>/<component>.log

<OFSAAI Installed Directory>/bdf/logs/20130313/CorrespondentBankProfile.log

When SQL*Loader is the loading mechanism, as shown below, there are additional log files containing log output from the SQL*Loader utility named the same as the component's log file with "_N" extensions (where N is an integer).

For example:
<OFSAAI Installed Directory>/bdf/logs/20130313/CorrespondentBankProfile_0.log
<OFSAAI Installed Directory>/bdf/logs/20130313/CorrespondentBankProfile_1.log

When an external table is used as the DIS file loading mechanism, there are additional log files containing log output from the external table utility. The log files are named the same as the external table being loaded. The name of the external table is the name of the table being loaded with a prefix of "DIS_". For example, when loading the ACCT table, the external table log file will be:

<OFSAAI Installed Directory>/bdf/logs/20130313/DIS_ACCT.log

Parameters

Parameters in TBAML Datamaps are specified as elements in an XML file. The XSD containing a description of these elements can be found in the following directory:
<OFSAAI Installed Directory>/bdf/config/ParameterSet.xsd
The Parameter element defines a parameter and its value, and contains the following attributes:
  • name - The name of the parameter.
  • type - The data type of the parameter. Valid values are STRING, REAL, INTEGER, BOOLEAN, FILE, and CLASS.
  • value - The value of the parameter, which must map the type of the parameter.
  • list - A boolean value specifying that the value is a single value (false - the default) or a comma separated list of values (true).

For example: <Parameter name="MinimumGeographyRisk" type="INTEGER" value="0"/>

<Parameter name="InternalAccountCodeList" type="STRING" value="IA,GL" list="true"/>

Note:

If the value of the parameter is a string containing characters which are not allowed in an XML attribute, then a CDATA element can be used as the element's text. For example:
<Parameter
name="PassThruExpressionSeparators"
type="STRING">
<![CDATA[~: \t/#-]]>
</Parameter>

Parameters in the main BDF.xml file should not be modified. Instead, any customizations to parameter values should be placed in the <OFSAAI Installed Directory>/bdf/config/custom/BDF.xml file. Parameters can be overridden at the component level by placing them in the custom/ <component>.xml file. Also, parameters can be overridden on the command line by passing the parameter name and value as parameters to the execute.sh script after the component name:

For example:
<OFSAAI Installed Directory>/bdf/scripts/execute.sh <component> [parameter
name=value]*
<OFSAAI Installed Directory>/bdf/scripts/execute.sh CorrespondentBankProfile
NumberOfThreads=4
When a given parameter is read by a component, the order of precedence for where the parameter value is taken from is as follows:
command line
<OFSAAI Installed Directory>/bdf/config/custom/<component>.xml
<OFSAAI Installed Directory>/bdf/config/<component>.xml
<OFSAAI Installed Directory>/bdf/config/custom/BDF.xml
<OFSAAI Installed Directory>/bdf/config/BDF.xml

Config

The config subdirectory contains configuration files.
  • <OFSAAI Installed Directory>/bdf/config/BDF.xml contains all default product configuration parameters. It should not be modified.
  • <OFSAAI Installed Directory>/bdf/config/install/BDF.xml contains all configuration parameters set at installation time (refer to the Installation Guide for more information).
  • <OFSAAI Installed Directory>/bdf/config/custom/BDF.xml contains any product configuration parameters that have been overridden for this installation. It is initially empty. Any changes to default product configuration parameters should be put here.

Individual components can have their own configuration file which overrides default product parameters. These files would be named using the following format:

<OFSAAI Installed Directory>/bdf/config/<component>.xml

For example:

<OFSAAI Installed Directory>/bdf/config/CorrespondentBankProfile.xml

Component configuration files in this directory are part of the product and should not be modified. If any parameters must be overridden at the individual component level, the component configuration file should be created in <OFSAAI Installed Directory>/bdf/config/custom.
  • The datamaps subdirectory contains XML files holding the data map definitions for components.
  • The derivations subdirectory contains SQL derivations for individual fields.
  • The queries subdirectory contains SQL queries for individual data maps.

BDF.xml Configuration Parameters

Table D-13 BDF.xml File Configuration Parameters

Type Description Description Example
MISCELLANEOUS NumberOfThreads The number of worker threads used by some components 4
MISCELLANEOUS SequenceBatchSize The batch size when retrieving sequence IDs for new records 100000
MISCELLANEOUS SourceSystem The default value for source system when one is not provided MTS
MISCELLANEOUS Currency The default value for issuing currency when one is not provided USD
MISCELLANEOUS Separator The delimiter that separates fields in data file records. ~
DB:Parameters related to database access. DB.Connection.Driver The JDBC driver class name. oracle.jdbc.O racleDriver
DB:Parameters related to database access. DB.Timeout The number of seconds to wait before timing out on a database connection attempt. 10
DB:Parameters related to database access. DB.NumRetries The maximum number of times to attempt to connect to a database before failing. 5
DB:Parameters related to database access. DB.MaxNumberOfDeadlocks The maximum number of times a deadlock is encountered during a JDBC insert or update operation, before an error is generated. 10
Directory:Parameters used to define directory locations. Directory.Inbox The input directory where the Oracle client will write DIS files. Date sub directories will be created in this directory where these files will be archived ../inbox
Directory:Parameters used to define directory locations. Directory.InternalData The directory where files generated by components will reside. This includes log files, error files, and any temporary processing files. ..
Log:Parameters used to configure the common logging module Log.Format Identifies the log formatting string.

%d[%t] %p -%m%n

Table D-14 BDF.xml File Configuration Parameters

Parameter Name Description Example
Log.UseDefaultLog Specifies whether the system uses the default log file for a component. The default log file has the name of the component and resides in a date subdirectory of the logs directory (in YYYYMMDD format). true
Log.SysLogHostName The host name of syslog for messages sent to syslog. hostname
Log.SMTPHostName The host name of the SMTP server for messages that processing sends to an e-mail address. hostname
Log.MaxSize The maximum size (in MB) of a log file before the system creates a new log file. 2000MB
Log.MaxIndex If a log file exceeds Log.MaxSize, this will be the maximum number of additional log files that are created (Component.log.1, Component.log.2, etc). 10
Log.TRACE.Enabled Indicates that trace logging is not enabled; true indicates enabling of trace logging. false
Log.TRACE.Location Specifies additional locations to send TRACE log messages to, other than the default log file (logs/YYYYMMDD/ Component.log). If the value is not provided, considers the default log location. false
Log.TRACE.Synchronous Specify whether logging for a particular level should be performed synchronously or asynchronously. false
Log.DIAGNOSTIC.Enabled DIAGNOSTIC logging is used to log database statements and will slow down performance. Make it true if needed. false
Log.DIAGNOSTIC.Location

Additional locations to send DIAGNOSTIC log messages to, other than the default log file (logs/YYYYMMDD/ Component.log).

If the value is not provided, considers the default log location.

 -
Log.DIAGNOSTIC.Synchrono us Specify whether logging for a particular level should be performed synchronously or asynchronously. false
Log.NOTICE.Enabled Indicates enabling of notice logging; false indicates that notice logging is not enabled. true
Log.NOTICE.Location Specifies additional locations to send NOTICE log messages to, other than the default log file (logs/YYYYMMDD/ Component.log). If the value is not provided, considers the default log location. -
Log.NOTICE.Synchronous Specify whether logging for a particular level should be performed synchronously or asynchronously. false
Log.WARN.Enabled Indicates enabling of warning logging; false indicates that warning logging is not enabled. true
Log.WARN.Location Specifies additional locations to send WARN log messages to, other than the default log file (logs/YYYYMMDD/ Component.log). -
Log.WARN.Synchronous Specify whether logging for a particular level should be performed synchronously or asynchronously. false
Log.FATAL.Enabled Indicates enabling of Fatal logging; false indicates that fatal logging is not enabled. true
Log.FATAL.Location Specifies additional locations to send FATAL log messages to, other than the default log file (logs/YYYYMMDD/ Component.log). -
Log.FATAL.Synchronous Specify whether logging for a particular level should be performed synchronously or asynchronously. false
Load:Parameters used to configure common Loading data - -
Load.FullRefresh For DIS files defined as Overwrite, whether to fully replace FCDM tables with the contents of the DIS file (true) or to treat the DIS file as a delta (false) True
Load.BatchSize The batch size when loading data. 5000
Load.Direct Specifies whether to use direct path loading (TRUE) or conventional path loading (FALSE). false
Load.Unrecoverable Specifies whether a direct path load does not use redo logs (TRUE) or uses redo logs (FALSE). false
Load.Partitioned Specifies whether a direct path load uses the current date partition (TRUE) or any partition (FALSE). false
Load.SkipIndexes Specifies whether a direct path load skips index maintenance (TRUE)or maintains indexes (FALSE). If set to TRUE, rebuilding of indexes must occur after running the DataMap XML. false
Load.DoAnalyze Specifies whether to run a stored procedure to analyze a database table after loading data into it. true
Load.AnalyzeType Specifies the type of analyze statistics has to perform if Do Analyze has a value of True. DLY_POST_ LOAD
Load.LogRecordInterval Specifies how often to log a message saying how many records a particular thread has inserted/updated, 1000
Load.MaxErrorRate Specifies the percentage of invalid records to allow before exiting with an error. For example, a value of 10 allows 10 percent of records to be invalid before exiting with an error. A value of 0 allows no invalid records. A value of 100 allows all invalid records. 100
Load.RecordQueueSize Specifies the number of records the query reader thread will write to a database writer thread queue before waiting for the reader thread to catch up. Higher values will require more memory usage. 100
Load.SkipIndexesErrorCode Specifies a database error code that occurs in the log file when skipping index maintenance. 26025
Load.IndexParallelLevel Specifies the parallel level of an index rebuild (that is, number of concurrent threads for rebuilding an index). 1
Load.DataErrorCodes Specifies a comma-separated list of database error codes that indicate data level errors , such as data type and referential integrity. This results in rejection of records with a warning instead of a fatal failure.

1,1400,1401,1

407,1438,172

2,1840,1841,2

291,2359,183

9,1847,12899
Load.ParallelLevel Specifies the level of parallelization to apply when loading data from a set of source tables to a target table. 8
Load.WriteErrorFiles Whether to check a DIS file for errors before loading as an external table (true) or not (false) True
DIS:Parameters related to processing DIS files - -
DIS.Source

The mechanism used to load DIS data.

FILE: DIS files will be provided and will be loaded using SQL*Loader processes running on the application server.

FILE-EXT:DIS files will be provided and will be loaded using external tables with the DIS files accessed directly by the database.

FSDW:DIS data will be obtained from database tables in the FSDW.

 FILE
DIS.ArchiveFlag Whether DIS files will be archived to a date subdirectory (true) or not (false). True
DIS.BufferSize The size in KB of the byte buffer used to read in DIS file records. 100
DIS.InputFileCharset The character set of the DIS files. Note that output data is always written in UTF8, this parameter just allows the DIS files to be in a different character set. -
DIS.Default.Check.Requirem ent Whether to check for mandatory fields on DIS records (true) or not (false). True
DIS.Default.Reject.Requirem ent Whether to reject DIS records for failing a mandatory field check(true) or to log a warning and attempt to load the record (false). True
DIS.Default.Check.Domain Whether to check that a DIS field has a valid domain value (true) or not (false). True
DIS.Default.Reject.Domain Whether to reject DIS records that fail a domain check (true) or not (false). True
DIS.Default.Check.Length Whether a DIS field should be checked for a valid length (true) or not (false). True
DIS.Default.Reject.Length Whether to reject DIS records that fail a length check (true) or not (false) True
DIS.Default.Check.Threshold Whether a DIS field should be checked that it is within an acceptable threshold (i.e. greater than 0) (true) or not (false). True
DIS.Default.Reject.Threshold Whether to reject DIS records that fail a threshold check (true) or not (false). True
DIS.Default.Check.Lookup Not currently supported. True
DIS.Default.Reject.Lookup - Not currently supported True
Parameters used by queries defined in the data maps: - -
MinimumGeographyRisk

Defines what is considered High Risk For the Account Profile attributes related to High Risk Geography , such as Incoming High Risk Wire Count.

Processing compares this parameter using a strict greater-than operation.

 0
AccountInactivityInMonths

Specifies the number of months that processing aggregated to determine whether an account is inactive. If the sum of trades and transactions over this number of months is <= 3, the account is considered inactive. This setting can impact the Escalation in Inactive Accounts scenario.

The default value is six months.

 6
TransactionsReversalLookba ckDays This parameter controls how many days of transactions to look across. Verify whether the new data contains reversals of prior transactions. 7
LowPriceSecurityThreshold Defines Low Priced in the base currency for the Account Profile attributes named Low-Priced Equity Range # Opening Trade Count. Processing compares the value of this parameter to the Trade table's Last Execution Price-Base. 5000
CommissionEquityPercentU pperLimit Defines the upper limit for Commission Versus Average Daily Equity Percentage in Account Profile Calculation. 5
TurnOverRateUpperLimit Defines the upper limit for Total Turnover Rate in Account Profile Calculation. 5
BankCodeListWithIA

Defines the List of Financial Institution Identifier Types, these are type of unique identifiers which are used to represent the financial institutions.

This parameter also contains IA (Internal Account Identifier)to be used in datamaps and is mainly used in

Correspondent Bank related datamap derivations. Below are the

list of examples

  • BIC: Bank Identifier Code (BIC)
  • CHU:CHIPS Participant User Identifier
  • CO:Corporate Identifier
  • CHP:CHIPS Participant Identifier
  • FED:Federal Reserve Routing (ABA) Number
  • CU:Customer Identifier
  • GL:General Ledger Account
  • IA:Internal Account Identifier

BIC,FED,CHP

,CHU, DTC,CDL,EP N,KID, CBI,CSN,OTF

,BLZ,I BAN,ABLZ,B SB,CP AP, SDIC, HEBIC, BCHH, NSC, IFSC, IDIC, PNCC, RCBIC, UKDSC,

Swiss BC, SwissSIC,IA
BankCodeList

Defines the List of Financial Institution Identifier Types, these are type of unique identifiers which are used to represent the financial institutions excluding Internal Account (IA).

This parameter does not contain IA (Internal Account

Identifier)to be used in datamaps and is typically used to derive financial institutions. Below are the list of examples

  • BIC: Bank Identifier Code (BIC)
  • CHU:CHIPS Participant User Identifier
  • CO:Corporate Identifier
  • CHP:CHIPS Participant Identifier
  • FED:Federal Reserve Routing (ABA) Number
  • CU:Customer Identifier
  • GL:General Ledger Account

BIC,FED,CHP

,CHU, DTC,CDL,EP N,KID, CBI,CSN,OTF

,BLZ,I BAN,ABLZ,B SB,CP AP, SDIC, HEBIC, BCHH, NSC, IFSC, IDIC, PNCC, RCBIC, UKDSC,

SwissBC, Swiss SIC
IdRiskWinLevel

Defines the Risk level to calculate Effective Risks for internal parties (Account/ Customer).

For example: Account 1234 has an Effective Risk of 5, IdRiskWinLevel can be set by the client. If the party identifier effective risk is greater than the set IdRiskWinLevel, then the party identity risk wins compared to fuzzy matcher (Party Name Risk). If not, fuzzy matcher wins.

 1
InternalAccountCodeList

Codes to define types of Internal Entities with client, for example:

  • IA:Internal Account Identifier
  • GL:General Ledger Account
 IA,GL
ExternalEntityCodeList

Codes to define types of External Entities with client, for example:

  • XA: ExternalAccount Identifier
  • CO:Corporate Identifier
  • DL:Driver License
  • IBAN:International Bank Account Number
 XA,CC,CO,D L,GM, GP,LE,MC,N D,NR, PP,SS,TX,AR, OT,IB AN
TrustedPairReviewReasonTe xt1 Defines the reason text1 for recommendation of canceling the Trusted Pair, due to increase in Risk of parties involved in trusted pair.

Risk of

<Party1> increased from<A> to

<b>
TrustedPairReviewReasonTe xt2 Defines the reason text2 for recommendation of canceling the Trusted Pair, due to increase in Risk of parties involved in trusted pair.

Risk of

<Party2> increased from<C> to

<D>
CorporateActionLookBackD ays This parameter determines the how many days trades to look back from the Corporate Effective Date. 7
DealNearTermMaturityDays

Defines the maximum number of days between the End Date and Trade Date.

This helps to calculate Structured Deals Initiated w/ Near-Term Exp. In Customer Profile/ Institutional Account Profile.

 7
ProfitLossUpperLimit Helps determine how much a security must move by the end of the day to be considered a win or loss. If the security moves by less than a specified percentage, processing does not count it either way. If it moves by this percentage or more, it counts as a win or a loss, depending on whether the movement was beneficial to the account that made the trade. 5
HouseholdTurnOverRateUp perLimit Defines the upper limit for Total Turnover Rate in Household Profile Calculation. 10000
HouseholdCommissionEquit yPercentUpperLimit Defines the upper limit for Commission Versus Average Daily Equity Percentage in Account Profile Calculation. 10000
OptionTradeAmountRange1 OptionTradeAmountRange2 OptionTradeAmountRange3 OptionTradeAmountRange4 OptionTradeAmountRange5 OptionTradeAmountRange6

Define the lower bound of each range for the Account Profile attributes named Options Range # Opening Trade Count.

Processing compares each parameter to the Trade table's Last Principal Amount- Base.

Each range is from the lower bound entered here to the lower bound of the next range.

 -
EquityTradeAmountRange1 EquityTradeAmountRange2 EquityTradeAmountRange3 EquityTradeAmountRange4 EquityTradeAmountRange5 EquityTradeAmountRange6

Define the lower bound of each range for the Account Profile attributes named Equity Range # Opening Trade Count.

Processing compares each parameter to the Trade table's Last Principal Amount- Base.

Each range is from the lower bound entered here to the lower bound of the next range.

 -

LowPricedEquityTradeAmou ntRange1

LowPricedEquityTradeAmou ntRange2

LowPricedEquityTradeAmou ntRange3

LowPricedEquityTradeAmou ntRange4

LowPricedEquityTradeAmou ntRange5

LowPricedEquityTradeAmou ntRange6

Define the lower bound of each range for the Account Profile attributes named Low-Priced Equity Range # Opening Trade Count. Processing compares each parameter to the Trade table's Last Principal Amount-Base.

Each range is from the lower bound entered here to the lower bound of the next range.

 -

MutualFundTradeAmountRange1

MutualFundTradeAmountRange2

MutualFundTradeAmountRange3

MutualFundTradeAmountRange4

MutualFundTradeAmountRange5

MutualFundTradeAmountRange6

Define the lower bound of each range for the Account Profile attributes named Mutual Fund Range # Opening Trade Count. Processing compares each parameter to the Trade table's Last Principal Amount-Base.

Each range is from the lower bound entered here to the lower bound of the next range.

 -
UnrelatedWhenOffsetAccountIsNull

This parameter is used to assign unrelated party code as "J" in the BackOfficeTransaction table, If OFFST_ACCT_INTRL_ID is null and UnrelatedWhenOffsetAccountIsNull is "Y",

If OFFST_ACCT_INTRL_ID is null and UnrelatedWhenOffsetAccountIsNullis "N", then unrelated party code is NULL.

 Y

Datamap Configuration File

Oracle clients can modify the BDF.xml file under the bdf/config/custom folder to override default settings that the system provides. You can also reapply any modifications in the current BDF.xml file to the newer BDF.xml file.

Override any settings in BDF.xml by placing the modifications in BDF.xml under the bdf/config/custom folder.

During installation, the following parameters are configured by the installer:
  • AccountTrustFromCustomer
  • DefaultJurisdiction
  • UseTaxidForUnrelatedPartyCode
  • BaseCountry
  • ProcessForeignFlag
  • ProcessBankToBank
  • ProcessTransactionXRefFlag
  • TrustedPairRiskReviewFlag

These parameters are stored in the following file:

<OFSAAI Installed Directory>/bdf/config/install/BDF.xml

Parameters DefaultJurisdiction and BaseCountry are defined in the InstallConfig.xml file during Silent Installation. Refer to the Installation Guide for more information.

The Installer sets the default value for other parameters as follows:
  • <Parameter name="AccountTrustFromCustomer" type="STRING" value="Y"/>
  • <Parameter name="DefaultJurisdiction" type="STRING" value="AMEA"/>
  • <Parameter name="UseTaxidForUnrelatedPartyCode" type="STRING" value="Y"/ >
  • <Parameter name="BaseCountry" type="STRING" value="US"/>
  • <Parameter name="ProcessForeignFlag" type="STRING" value="N"/>
  • <Parameter name="ProcessBankToBank" type="STRING" value="N"/>
  • <Parameter name="ProcessTransactionXRefFlag" type="STRING" value="Y"/>
  • <Parameter name="TrustedPairRiskReviewFlag" type="STRING" value="N"/>

To change the default value of these parameters, before running ingestion, go to <OFSAAI Installed Directory>/bdf/config/install/BDF.xml and change the value to 'Y' or 'N' as needed.

Table D-15 Datamap Configuration Parameters

Property Name Description Example
DB.Connection.URL Database URL for JDBC connections made by components. The content and format of this value is specific to the database vendor and the vendor database driver. jdbc:oracle:thin:@solit aire.mantas.com:1521: D1O9L2
DB.Connection.Instance Database instance to connect to on the database servers. Typically, the instance name matches the database name portion of the DB.Connection.URL. D1O9L2
DB.Connection.Password Password that Java Ingestion components use when connecting with the database. This is set by executing bdf/scripts/changepassword.sh -
DB.Schema.MANTAS Schema name for the Oracle ATOMIC database schema. accesses the ATOMIC schema when allocating sequence IDs to ingested records. ATOMIC
DB.Schema.MARKET Schema name for the ATOMIC database schema. Data Management stores market data related records in the ATOMIC schema. ATOMIC
DB.Schema.BUSINESS Schema name for the ATOMIC database schema. Data Management stores business data related records in the ATOMIC schema. ATOMIC
DB.Schema.CONFIG Name of the configuration schema owner. REVELEUS
DB.Schema.CASE Name of the ATOMIC schema owner. ATOMIC
DB.Alg.Connection.User Database user for running Behavior Detection post- processing jobs. ATOMIC
DB.Alg.Connection.Passwor d Password for the DB.Alg.Connection.User. -

There are also configuration files for individual components that are delivered as part of the product:

<OFSAAI Installed Directory>/bdf/config/<component>.xml

And can also be created in the following directory:

<OFSAAI Installed Directory>/bdf/config/custom/<component>.xml