8      Metadata Lineage

This section helps you to navigate through the Metadata Lineage and guides you in tracing the Metadata Lineage tools such as Metadata Browser, Metadata Report and Data elements.

Metadata Browser

This section helps you to navigate through the Metadata Browser and guides you in tracing the source of the metadata. The Metadata Browser function allows you to view and analyze all aspects of the metadata used in the OFSAAI. It provides extensive browsing capabilities of metadata, helps in tracking the impact of changes to metadata, and trace through to the source of originating data.

Metadata Browser (Object and Application View) provides a common repository of metadata objects created in OFSAAI and applications hosted in OFSAAI. Using this view, you can identify the usage of base objects in higher-level objects and the mapping of Objects to Application, thus enabling traceability. It also allows you to view the data flow and the workflow of the application and understand the usage of objects within the application.

The new visualization of Metadata Browser (MDB) supports the Application view and Object view. In the Application view, you can browse through the metadata created using the applications hosted in OFSAAI. In the Object view, you can view the metadata created in OFSAAI.

To access the Metadata Browser (Object and Application Views), your role must be mapped to the SCR_MDB function.

Analysts review the metadata used for a particular report schedule to verify the data. Data verification may require looking for metadata used in a given schedule or schedules in which particular metadata is used. Data Analysts and Reporting Analysts perform the report verification. Metadata refers to business measures, hierarchies, data sets, derived entities used for a given schedule.

Reporting Metadata

To use MDB for schedule-wise metadata, and to use MDB for metadata wise schedule, identify the metadata used, perform the following steps:

1.     You can verify the data for related data elements in results using this information. Navigate to Lineage, select Metadata Browser, select Catalog of Objects, select OFSAA Metamodel, select Reporting Metadata, and then select Reports. The MDB Reporting Metadata screen is displayed.

Figure 122: MDB - Reporting Metadata Page

2.     Click the object view FR2028D to view the list of schedules. The Reporting Metadata Schedule View page is displayed.

Figure 123: MDB - Reporting Metadata - Schedule View

You can view the following information in the Schedule Details page:

§       Depends On: This section displays the metadata used in a given schedule.

§       Used In: This section displays the Reports in which this schedule is used.

§       Applications: This section displays the applications in which this schedule is used.

3.     Click Details to view the dependency and usage information such as the Object Name and the Object Type.

Figure 124: MDB - Reporting Metadata - Schedule View 1

4.     From the Schedule View page, click the Dependency tab to view the report tree structure.

Figure 125: MDB - Reporting Metadata Tree Structure Page

Starting from common metadata used across the application, you may want to know the list of reports or derived entities this metadata has used. Let us take an example of a measure. To identify how value is computed, perform the following steps to trace it back to the metadata.

Business Metadata

This section provides information on the Business metadata objects which include Base Metadata and Derived Metadata.

Base Metadata

The following are the steps to perform to view the Base metadata details. For example, Measures.

1.     To view the measures, navigate to Catalog of Objects, select OFSAA Metamodel, select Business Metadata, select Base Metadata, and then select Measures.  The MDB Business Metadata page is displayed.

Figure 126: MDB - Business Metadata - Measure View Page

2.     Click the Measure that you wish to view. The MDB Business Metadata Measure Details page is displayed.

Figure 127: MDB - Business Metadata Measure Details Page

You can view the following information on this page:

§       Measure Properties: It provides information on the properties of Business measures. For example, aggregation function, Measure Data Type, Business Exclusions, Filter, and Rollup Type.

§       It depends on: This section displays all the object names and their types, such as Entities, Columns, and so on.

§       Used In: This section displays the Objects in which this schedule is used.

§       Applications: This section displays the applications in which this schedule is used.

3.     Click Details to view the measure dependency and usage information.

Figure 128: Measure Dependency and Usage Details Page

4.     From the Measure Details page, click the Dependency tab to view the measure tree structure.

Figure 129: Business Metadata Measure Tree Page

NOTE:   

The similar steps as mentioned in this section are applicable for other metadata such as Business Metadata (Hierarchies, Measures, Variables, and so on) and Derived Metadata (Dimensions, Filters, and so on), Process Metadata (Process, Rules, and so on) and Data Foundation (Target Model, Sources, Connectors, and so on).

Derived Metadata

The following are the steps to perform to view the Derived Metadata details. For example, Derived Entities.

1.     To view the schedule-wise derived entities, navigate to Catalog of Objects, select OFSAA Metamodel, select Business Metadata, select Derived Metadata, and then select Derived Entities.

Figure 130: MDB - Business Metadata – Derived Entity Page

2.     Click the Derived Entity that you wish to view. The Derived Entity Details page is displayed.

Figure 131: Derived Entity Details Page

You can view the following information on this page:

§       Derived Entity Properties: It provides information on properties of derived entities, such as Source Type, Aggregate Flag, and Materialized View.

§       It depends on: This section displays all the object names and their types, such as Dataset, Hierarchy, and so on.

§       Used In: This section displays the Objects in which this schedule is used.

§       Applications: This section displays the applications in which this schedule is used.

3.     Click Details to view the derived entity dependency and usage information.

Figure 132: Derived Entity Dependency and Usage Page

4.     From the Derived Entity Details page, click the Dependency tab to view the Derived Entity tree structure.

Figure 133: Derived Entity Tree Structure Page

For more information about the Metadata and its usage, see the OFSAA Metadata Browser User Guide.

Metadata Report Utility

The Metadata Export Utility helps you to export OFSAA metadata into Excel Sheet. This feature helps to get a view of OFSAA metadata and its dependencies. It is a template-based approach where you create templates and select Metadata Objects that must be extracted. The extraction process is supported only for Excel Sheet. While defining the template, you are expected to have prior knowledge of the OFSAA Metadata objects that are relevant from this application point of view.

Prerequisites

The following executions must be performed before using the Metadata Export Utility:

1.     Before executing MDB Publish and Data Elements Wrapper Batch, ensure the following:

a.     Tablespace Requirement:

i.       Ensure that the USERS tablespace have a minimum of 150 GB available

ii.     Ensure that the TEMP tablespace is a minimum of 45 GB available

b.     Execute the following Gather Stat command for the mentioned tables:

BEGIN

DBMS_STATS.GATHER_TABLE_STATS(USER, 'TABLE_NAME');

END;

i.       Atomic Schema:

—    FSI_M_CELL_DETAILS

—    FSI_DE_SEEDED_DIMENSIONS

—    FSI_DE_TABLE_APPLICATION_MAP

—    FSI_DE_PP_TABLE_LIST

—    FSI_DE_METADATA_SEEDED_VW_MAP

—    FSI_DE_PP_TABLE_REPORT_MAP

ii.     Config Schema:

—    AAI_OBJECT_B

—    AAI_OBJECT_TL

—    AAI_DMT_DEFINITION

—    AAI_DMT_DEF_SOURCE_ENTITY

—    AAI_DMT_MAPPING_DETAILS

—    PR2_RULES_B

—    PR2_RULE_MAP

—    PR2_RULE_OBJECT

—    PR2_RULE_OBJECT_MEMBER

—    PR2_OBJECT_TL

—    PR2_OBJECT_TRACE

—    BATCH_MASTER

—    BATCH_TASK_MASTER

—    BATCH_PARAMETER_MASTER

—    METADATA_MASTER

—    METADATA_ELEMENT_MASTER

—    METADATA_LOCALE_MASTER

—    METADATA_TYPE_MASTER

—    METADATA_ATTRIBUTE_MASTER

2.     MDB Publish: Execute the batch, INFODOM_MDB

3.     After Executing MDB Publish and Data Element Wrapper Batch, ensure the following:

a.     Execute the following Gather Stat command for the mentioned tables:

BEGIN

DBMS_STATS.GATHER_TABLE_STATS(USER, 'TABLE_NAME');

END;

i.       Atomic Schema:

—    FSI_DE_REPORT_LINEAGE_BASE

—    FSI_DE_METADATA_TGT_MEMBER

—    FSI_DE_METADATA_SRC_MEMBER

—    FSI_DE_REPORT_TARGET_MEMBER

—    FSI_DE_REPORT_SOURCE_MEMBER

4.     Logs: MDB logs are generated under deployed area /Context_Name/logs/MDB_XXXX.log

5.     Data Elements Wrapper Execution: After MDB Publish is completed successfully with the message “Metadata publishing is finished.” in the /Context_Name/logs/MDB_XXXX.log, you must execute the Data Elements Utility with the following seeded batch to get the Data Lineage for each Metadata in OFSAA:

<INFODOM>_POP_DATA_ELEMENTS_USFED

This execution requires adequate tablespace. Ensure that your Atomic Schema is having enough tablespace in TEMP and USERS.

Parameters used in <INFODOM>_POP_DATA_ELEMENTS_USFED Batch

The batch can be executed in different modes according to each requirement. The following are the parameters used for executing the batch.

The default parameters used in the <INFODOM>_POP_DATA_ELEMENTS_USFED batch are:

Table 33: Task1 (METADATA PARSER)

Table 34: Task2 (REPORT PARSER)

Execution Types for METADATA Parsing in <INFODOM>_POP_DATA_ELEMENTS_USFED Batch

6.     Full METADATA Parsing [Default Mode] (if the P_FULL_PARSE parameter is ‘Y’, then the parsing happens for entire METADATA and Run Elements for the Run(s) enabled in FSI_DE_POP_RUN_LIST table in the Atomic Schema).

7.     Incremental METADATA Parsing [Optional Mode. Batch Parameter to Be Modified] (if the P_FULL_PARSE parameter is ‘N’, then the parsing happens for changed METADATA and Run Elements for the Run(s) enabled in FSI_DE_POP_RUN_LIST table in the Atomic Schema).

You can edit the parameters by accessing the Batch Maintenance screen.

a.     Log in to Oracle Financial Services Analytical Applications interface with your credentials.

b.     Navigate to Regulatory Reporting for US Federal Reserve, select Process and Operations, select Operations and then select Batch Maintenance

c.     Select Batch Name (<INFODOM>_POP_DATA_ELEMENTS_USFED)

d.     (OPTIONAL) Select Task1 and click the Edit button. The Edit Task Definition Window is displayed.

e.     Modify the Parameter List field as applicable.

The values must be in single quotes and comma-separated for each value. Follow the same order as in the table.

Execution Types for REPORT Parsing in <INFODOM>_POP_DATA_ELEMENTS_USFED Batch:

8.     US FED Jurisdiction REPORT Parsing [Default Mode] (if the P_JURISDICTION parameter is ‘USFED’, then the parsing happens for US FED Reports enabled in FSI_DE_POP_REPORT_LIST table in the Atomic Schema).

Even if the P_JURISDICTION parameter in <INFODOM>_POP_DATA_ELEMENTS_USFED Batch is loaded, the Dashboards which get parsed depend on the FSI_DE_POP_REPORT_LIST table in the Atomic Schema.

9.     All Jurisdictions REPORT Parsing [Optional Mode. Batch Parameter to Be Modified] (if the P_JURISDICTION parameter is NULL, that is, (‘’) or two Single Quotes, then the parsing happens for entire Reports enabled in FSI_DE_POP_REPORT_LIST table in the Atomic Schema).

You can edit the parameters by accessing the Batch Maintenance screen.

a.     Log in to Oracle Financial Services Analytical Applications interface with your credentials.

b.     Navigate to Regulatory Reporting for US Federal Reserve, select Process and Operations, select Operations and then select Batch Maintenance

c.     Select Batch Name (<INFODOM>_POP_DATA_ELEMENTS_USFED)

d.     (OPTIONAL) Select Task2 and click the Edit button. The Edit Task Definition Window is displayed.

e.     Modify the Parameter List field as applicable.

The values must be in single quotes and comma-separated for each value. Follow the same order as in the table.

Enabling Run for METADATA Parsing

Every execution for METADATA Parsing requires a minimum one Run to be enabled in the FSI_DE_POP_RUN_LIST table in the Atomic Schema. By default, RGRNUSFED is enabled.

Table 35: Run Names for Metadata Parser

Excluding Irrelevant Data Flows from Lineage Reports

For each Run, some of the Data Mappings can be functionally irrelevant. For these cases with respect to any Run, the customer can opt for removing these Data Flow from Lineage Reports as an exclusion by inputting the same in the FSI_DE_RUN_FLOW_REMOVAL table.

Enabling Reports for REPORT Parsing

Every execution for REPORT Parsing requires a minimum one Report to be enabled in the FSI_DE_POP_REPORT_LIST table in the Atomic Schema. By default, the following Reports are enabled for US FED Jurisdiction.

Table 36: Report Codes for Report Parser

By default, All Dashboards are enabled and if you wish to parse particular Dashboards, modify the FSI_DE_POP_REPORT_LIST table in the Atomic Schema by enabling/disabling the “Include Report Column”.

Executing SELECTED tasks of <INFODOM>_POP_DATA_ELEMENTS_USFED Batch

By default, the <INFODOM>_POP_DATA_ELEMENTS_USFED Batch contains both the tasks, that is, METADATA Parsing and REPORT Parsing. You can use the platform feature of the EXCLUDE / INCLUDE Batch Task for the Optional execution of required tasks.

Topics:

·        Verifying Logs

·        Validating Lineage Outputs 

Verifying Logs

Data Elements logs are generated in Atomic Schema under the FSI_MESSAGE_LOGS table.

Table 37: Data Element Logs

Validating Lineage Outputs

In Atomic Schema, you must verify that data is present in the following tables and ensure that the table is populated:

·        FSI_DE_RUN_LINEAGE_METADATA

·        MDR_LINEAGE_METADATA

·        FSI_DE_REPORT_LINEAGE_BASE

It is recommended that the following SQL statement must be executed in Config Schema if this INDEX is not created:

CREATE INDEX index_mdr_mod_parent_child
CREATE INDEX index_mdr_mod_parent_child
ON mdb_object_dependencies (parent_object_def_id,child_object_def_id)
COMPUTE STATISTICS
/

User Access

The following user groups are pre-seeded in the component that helps you get access to the Metadata Report Extract screen.

1.     MDR View Group: To see Metadata Report Extract with View permissions.

2.     MDR Owner Group: To create templates in Metadata Report Extract.

Create and Export Metadata Report Templates

Perform the following steps to create and export the Metadata Report Templates:

1.     Navigate to Lineage and then select Metadata Report.

Figure 134: Metadata Report Template Summary page

2.     Click Add icon, in Summary screen, to create a new Metadata Report Template.

Figure 135: Add Metadata Report Template page

3.     Provide the Name and Description for the new template in the Template Definition page.

Figure 136: Template Definition window

4.     Select the desired object from the Object Type dropdown to be exported.

5.     Individual report generates only the basic properties of the object selected, that is, name and description. Relational report generates detailed information up to the Entities level if Dependencies is chosen; and up to the Staging Columns level, if Data Lineage is selected along with Dependencies.

6.     Dependencies: Metadata object is dependent on several other metadata objects. Metadata object is also used (that is, consumed) in several other metadata objects. Dependency or usage tree can be of any depth. For example, a rule can be dependent on a hierarchy, business processor, and dataset. Further, each of these metadata objects can be dependent on other metadata objects. Metadata Export Utility exports all the dependent or used metadata objects for all paths in the dependency or usage tree if this option is selected.

7.     Lineage: Data is loaded from source systems to staging and then moved across to processing/reporting. Lineage traces the data element as it moves across different layers of OFSAA: staging, processing, and reporting. Metadata Export Utility exports the lineage of each of the reporting area data element that is identified by dependencies.

8.     For Individual Report: In the Export Options, do not select Dependencies or Data Lineage.

Figure 137: Object Types window

9.     The exported sample report for Individual is as follows:

Figure 138: Individual Sample Report

For Relational Report: In the Export Options, select Dependencies.

Figure 139: Export Options

10.  The exported sample report for Relational is as follows:

Figure 140: Relational Sample Report

11.  The first sheet shows the different Paths and their Dependencies until the Entities level. Select the required Path sheet at the bottom to view the dependencies.

Each path tells how the dependency/usage is derived from dashboard to entity or vice versa involving various OFSAA object types like Derived Entity, Hierarchies, Datasets, Measures, and so on.

These paths are generated by the system using data already published in MDB dependency tables as part of the OFSAA MDB object publish.

For every dependent object type displayed in each path sheet, the following columns are displayed:

§       Object type name

§       Object type description

§       One or many Object-specific properties (optional)

12.  For example: In Path1, Dashboard is the first Object type, the dependencies generated are Dashboard Name, Dashboard Description, and Dashboard properties: Dashboard Country, Dashboard Regulator and so on. Similarly, Report is the next Object type in Path1 and the dependencies generated are Report Name, Report Description, Views Name, Views Description, View Display Format and so on. Then followed by Hierarchy Objects name, description, and properties up to the Entities level.

Figure 141: Path 1 Object Type

The Usage sample report (generated by default when Dependencies is selected) is as follows:

Figure 142: Usage Sample Report

The first sheet shows the different Paths and their Usage until the Dashboard level. Select the required Path sheet at the bottom to view the Usage.

Figure 143: Patch Selection Sheet

13.  Select Data Lineage in Template Definition and then select Choose Object Type to export the lineage details up to the Staging Columns level.

Data Lineage can be selected only if Dependencies option is chosen.
The minimum memory settings to run lineage reports should be
export JAVA_OPTS="-Xms1024m -Xmx8192m"

Figure 144: Object Type Selection

Data Lineage is generated as a separate sheet in the generated Relational report along with the Dependencies. Select the Lineage sheet to view the Data Lineage (up to Staging column level).

Figure 145: Lineage Details

14.  Select Filter Objects to see the selected objects.

Figure 146: Template Definition Filter Objects window

15.  Select one Filter Object from the Available Objects and click  to add a Selected Object. Select one Selected Object from the Available Objects and click  to remove a Filter Object.

Figure 147: Object Selection window

16.  When the object list is huge, use the Search option as shown above. Type first three letters of the Filter Object name and the relevant Filter Objects are displayed.

You can type the complete Filter Object name to select and add to the Selected Objects.

Figure 148: Object Search Option window

17.  Select the Lineage Properties required to be generated.

The following Lineage Properties (columns) are available in the Metadata Report Screen.

Table 38: Lineage Properties

Figure 149: Lineage Properties window

18.  Review the Template Definition once and click Save.

Figure 150: Template Definition Review window

19.  Click Return to go to the Summary page.

Figure 151: Template Definition Summary-Export to Excel page

20.  Select a template in the Template List in the Summary screen and click Export to Excel to export the desired objects in Excel Sheet format.

MDB Publish must be triggered before executing the Export to Excel option.

21.  Select a template in the Template List in the Summary screen and click Export to XML to export the desired objects in XML format.

MDB Publish must be triggered before executing the Export to XML option.

Figure 152: Template Definition Summary-Export to XML page

22.  The Report Generation function is an asynchronous action and to check the status of the export function, use the Refresh option in the Summary screen.

Figure 153: Template Definition Summary- Refresh page

§       For Excel Export, the following are the Status values:

§       Not Started: The Report Generation is yet to start, but the function has triggered the action in the background.

§       Ongoing: The Report Generation is started and in process.

§       Completed: The Report Generation is completed and ready to view or download.

§       Failed / Partially Completed / No Path Found: The Report Generation encountered an issue and the process is partially completed or failed.

The export logs are generated and placed in the path /Context_Name/logs/MDB.log.
Log files give the following information:

a.     All Paths query

b.     Query for each path and if data present for this path

c.     Lineage query

d.     Status of excel output creation

e.     Exceptions and errors, if any

23.  Select a template in the Template List in the Summary screen and click Download to save a copy of the generated Metadata Report Templates excel sheet, after the export status shows as completed.

Figure 154: Template Definition Summary-Download page

User Access

The following user groups are pre-seeded in the component that helps you to get access to the Metadata Report Extract screen.

1.     MDR View Group: To see Metadata Report Extract with View permissions.

2.     MDR Owner Group: To create templates in Metadata Report Extract.

View Metadata Report Templates

Perform the following steps to view the Metadata Report Templates:

1.     Select a template in the Template List in the Summary screen.

2.     Click View icon to view the generated Metadata Report Templates excel report (after the export status shows as completed).

Figure 155: Template Definition Summary View Page

The Metadata Report Templates excel report is opened in view-only mode.

Modify or Edit Metadata Report Templates

Perform the following steps to edit or modify the Metadata Report Templates:

1.     Select a template in the Template List in the Summary screen.

2.     Click Edit icon to modify the generated Metadata Report Templates excel report (after the export status shows as completed).

 Figure 156: Template Definition Summary Edit page

Delete Metadata Report Templates

Perform the following steps to delete the Metadata Report Templates:

1.     Select a template in the Template List in the Summary screen.

2.     Click Delete icon to delete the Metadata Report Templates.

Figure 157: Template Definition Summary Delete page

Viewing Data Elements Summary

Select Data Elements Summary from the main navigation menu to view all the Data Elements.

 Figure 158: Data Elements Summary

By default, the page displays all the data elements.

Figure 159: Selection Panel

Click a row and the selection panel displays the selected entity and attribute.

Figure 160: Selected Entity

The tabs on the right can be used to view reports, schedules, and cells as shown in Figure 160, which are utilized for the selected data element.

Figure 161: Report or Schedule View

NOTE:   

For Viewing Data Elements and Viewing Data Elements Summary, Data Elements batch execution is required for the screen to function.