12.6.1 Pipeline

A pipeline is an embedded data processing engine that runs inside the application to filter, transform, and migrate data on-the-fly. Pipelines are a set of data processing elements called widgets connected in series, where the output of one widget is the input to the next element. Use the Pipeline canvas to create the model and execute the pipeline using widgets.

To create a model using pipeline designer, follow the subsequent steps:

  1. This icon shows the adding dummy node with no paragraph. Navigate to the Pipeline Designer page. Pipeline Canvas is displayed.
  2. Click on the Connector to display the widgets.
  3. Select Transaction Filtering from the list.
  4. Select a widget and add the widget to the pipeline canvas. For information on widgets, see the TF Pipeline Widgets table.
  5. From the pipeline canvas double click on the widget to open the widget details screen on the right side.
  6. In the widget screen under the Custom Parameters tab, click Copy to open the Clone Objects

    Window.

  7. Select the source model ID from the Clone Objects Window and select the version from which you want to clone the widget.

    Note:

    For the first model, select model ID as PROD.
  8. Click Copy. The TF Widget clone process begins. Once the cloning is completed, the current model ID and version will automatically be populated in the widget screen.
  9. Click Save to save the widget.
  10. Click Add the next widget and repeat from step 2 to step 8.

    Figure 12-22 Pipeline Canvas


    Pipeline Canvas

The TF Pipeline Widgets table gives information about TF pipeline widgets.

Table 12-5 TF Pipeline Widgets

Widget Description
Real time EDQ Opens EDQ homepage where users can log in to EDQ director and tune the EDQ rules of the project with respective model_id with version 0.
Swift Message Configurations Add/Edit/Remove Swift Configurations
Swift Message Parser Posts the selected messages (transactions) to the JMS queue
TF Simulation Report Displays the Summary of alerts and event hits for the current simulation run.
TF Simulation Data Displays the events generated for the current simulation run.
TF Production Data Displays the production events for which the simulation was carried out.
TF Swift Configurations Audit This report will show the list of all Swift Configuration changes with respect to Source Model Swift Configurations.

To execute the pipeline follow the subsequent steps:

  1. Click execute icon This icon shows the launch workspace.. Execute Pipeline window is displayed.

    Figure 12-23 Execute Pipeline Window


    Execute Pipeline Window

  2. Click Open from saved Parameter set? to import the template.
  3. If your not importing the template enter the execution Key and Value manually. Execution parameters are the filters to apply to production data for the simulation run. You configure these filters in FCC_TF_SIMULATION_FILTERS table. For more information on FCC_TF_SIMULATION_FILTERS, see OFS Sanctions Data Model Reference Guide.

    Note:

    Select the flow, which you want to execute Scoring, Training, and Experimentation. It displays all the keys defined for all the paragraphs in the notebook with a placeholder for providing the values.
  4. You can add new parameters using Add

    Figure 12-24 Add

    This icon shows the add new parameters in the execute pipeline window.
    .

    Note:

    If the parameter is not defined in the notebook, it will not be used for the execution. In case of multi select, if there are common parameters among the chosen scenarios, it will take the value based on the order of selection. that is first chosen scenario parameter will be taken.
  5. But if open from saved scenarios again (not on single go), then already added will get replaced by the newly added (same as what existed)
  6. Execution is performed based on selected link types. It filters out all the not required/unused parameters. And, all the unused parameters for the current execution are displayed with a

    warning This icon shows the warning of all the unused parameters for the current execution.. To view the only required parameters, click Show only required link.

  7. Click Reset This icon shows the reset option of the entered data. to reset the entered data.
  8. Click Delete This icon shows the delete option of the entered Key and Value. to delete the entered Key and Value. For example, refer to below Figure.
  9. This icon shows the model pipeline menu. Click Execute to initiate the execution. The widgets in pipelines are executed sequentially and

you can see icon on each widget for a successful execution. For individual widget execution details click the widget and click View Details.