4.4.3 Processing Data Using FDT and MDT

The following sections describe how Ingestion Manager processes trade-related data, orders and executions, and trades through the Firm Data Transformer (FDT).

FDT Process Flow

The following figure illustrates the FDT process flow:

Figure 4-2 Firm Data Transformer (FDT) Processing


Description of Figure 4-2 follows
Description of "Figure 4-2 Firm Data Transformer (FDT) Processing"

The FDT performs the following actions:
  • Processes all files that reside in the /data/firm/transform directory for the current date and batch.
  • Terminates automatically after processing files that it found at startup.
  • Ignores files that the system adds after processing begins; the system may process these files by starting FDT again, after exiting from the previous invocation.
Order and Trade Execution files are processed through the Firm Data Transformer (FDT). Before running runFDT.sh, Pre-processor has to be executed, using the following commands:
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDP.sh TradeExecution
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDP.sh Order
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDP.sh OpenOrder
During execution of the runFDT.sh script, the FDT performs the following actions:
  • Enriches data.
  • Produces summary records for orders and trades.
  • Calculates derived values to support detection needs.
  • Derives state chains (that is, order life cycle states, marketability states, and displayability states).
  • Provides data for loading into FSDM.

The system executes the FDT with the runFDT.sh script; the following provides a sample command: <OFSAAI Installed Directory>/ingestion_manager/scripts/runFDT.sh

When Ingestion Manager executes runFDT.sh, it places output files in the directories described in the following table.

Table 4-3 runFDT.sh Output Directories

Directory Description
/data/firm/transform Rollover data that processing saves for the next run of the FDT. Includes open and closed orders, old executions, old trades, old derived trades, lost order events, and lost trade execution events.
/logs/<yyyymmdd> Status and error messages.
/data/errors/<yyyymmdd> Records that the system was unable to transform.
/data/backup/<yyyymmdd> Backup of Pre-processed input files.
/data/firm/load Transformed output files for loading into the database.
After running run FDT, the system executes data loaders using the runDL.sh script; the following provides a sample command:
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDL.sh Order
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDL.sh OrderSummary
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDL.sh TradeExecution
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDL.sh Execution
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDL.sh Trade
<OFSAAI Installed Directory>/ingestion_manager/scripts/runDL.sh DerivedTrade

FDT processes are also available with BDF. To perform this action, you must execute the following datamaps in the order given:

  1. OpenOrderStage
  2. OrderStage
  3. TradeExecutionEventStage
  4. Scrty_TradeExecutionStageInsert
  5. Scrty_OrderStageInsert
  6. MktCntr_OrderStageInsert
  7. OrderStage_DQupdate
  8. TradeExecutionEventStage_DQupdate
  9. OrderStage_FDTupdate
  10. OrderStage_RmngQtupdate
  11. OrderSummary
  12. OrderSummary_OpenOrdrInsrt
  13. OrderSummary_QtyUpdate
  14. OrderStage_OpenOderUpd
  15. OrderSummary_Update
  16. OrderStage_OrdrSeqUpd
  17. OrderEvent_OrderStage
  18. Execution_NewEvents
  19. Execution_CancelAndReplace
  20. Execution_CancelEvents
  21. Execution_CorrectionEvents
  22. Trade_NewEvents
  23. Trade_CancelAndReplace
  24. Trade_CorrectionEvents
  25. Trade_CancelEvents
  26. Trade_DerivedTrade
  27. Trade_OrigSeqIDUpd
  28. Trade_ParentSeqIDUpd
  29. Trade_RplcngSeqIDUpd
  30. TradeExecutionEvent_Trade
  31. TradeExecutionEvent_Execution
  32. TradeExecutionEvent_CancelReplaceTrade
  33. TradeExecutionEvent_FirmRefTrade
  34. TradeExecutionEvent_MktRefTrade
  35. Trade_RefData
  36. Execution_Update
Trade related Data maps cannot run using Multi Batch Functionality.

Populating Summary Information for Market Data

As part of end of day processing, Market and Trade data summary information gets updated in the following path of the Java Utility: <OFSAAI Installed Directory>/ingestion_manager/scripts/ process_firm_summary.sh