This chapter provides information about Big Data processing supported in the Oracle Financial Services Data Foundation application.
Topics:
· Overview
OFSDF supports Big Data processing on HDFS (Hadoop Distributed File System) using Hive as Query Engine. The AAI application components such as SCD, H2H, and DQs provide equivalent Hive support. For more information, see Oracle Financial Services Advanced Analytical Applications Infrastructure User Guide Release 8.1.0.0.0.
In Big Data processing, OFSDF provides two types of deployment processes for Hive support. They are:
· Staging on Hive and Results on RDBMS
Topics:
· Workarounds in the Staging and Results on Hive
· List of Supported SCDs and T2Ts
· Executing Run through Rules Run Framework for Hive
In the Staging and Results on Hive deployment process, the Staging, and Results model elements are a part of Hive known as Hive Datadom. The config schema definitions used by AAI and certain metadata definitions that were part of Atomic schema now resides in a RDBMS schema known as Metadom. Refer to the below architecture diagram for the representation of this deployment process.
Figure 55: Data Foundation Big Data Architecture with Staging and Results on Hive
For the list of supported FSDF Hive Metadata definitions, refer to Run Chart, SCD Metadata, and Technical Metadata (Staging Source) documents at My Oracle Support.
NOTE |
Ensure that String values, which are null, must contain \n and then source to the Hive stage tables. |
In the Staging on Hive and Results on RDBMS deployment process, the Staging occurs on Hive, known as Hive Datadom, and the Results model elements are a part of RDBMS, known as RDBMS Datadom. In this deployment process, an additional software is required, which is Oracle Big Data SQL (For the software version information, see the Oracle Financial Services Analytical Applications Technology Matrix Release 8.1.1.0.0). Oracle Big Data SQL software resides on Hive and RDBMS Datadoms, and enables the user to create a link to Hive Stage tables as external tables in RDBMS. This software manages the data representation of the Hive tables in RDBMS Datadom and can be accessed as external table objects. As a result, enables the use of RDBMS infrastructure components of SCD, T2T, and DQs. Refer to the below architecture diagram for the representation of this deployment process.
Figure 56: Data Foundation Big Data Architecture with Staging on Hive and Results on RDBMS
This section provides information about the workarounds in the Staging and Results on Hive.
Topics:
Modifications in OFSDF Hive (Staging and Results on Hive deployment process) compared to OFSDF RDBMS are:
· Dim Dates Population
The Dim Dates functionality in Hive is similar to that of RDBMS. The difference exists in the date input format on the UI. In OFSDF Hive, the Date Range input format is, an executable textbox contains .sh file name followed by the FROM and TO dates.
For example:
load-dimdates-run.sh,20110101,20110105
The list of workarounds in OFSDF Hive (Staging and Results on Hive deployment process) is:
· The process of data access, modification, and representation is different in HDFS compared to that of RDBMS. Therefore, the mechanism for Data Protection also varies.
Workarounds:
§ Data Redaction
The process of enabling Data Redaction for Hive can be performed using CDH Manager (Cloudera Distribution Hadoop Manager). For more information, see the section Sensitive Data Redaction in the Cloudera Security Guide.
§ Right to Forget
The Right to Forget feature enhancement is planned for future releases. However, prior to loading in the Hive instance, Anonymization of Party PII data can be done at the source. This process provides the flexibility to secure the PII information, and in addition, to drop the PII upon a request from the Party.
· Mappers are supported using AM/HM screens in OFSDF (RDBMS). However, development of the underlying functionality for AM/HM feature in OFSDF Hive is planned for future releases. Therefore, the Mappers will be supported during future releases.
Workaround:
The Views defined on these Mappers in RDBMS are converted to Tables in Hive. Therefore, these Mappers must be loaded manually:
§ MAP_BAL_CAT_STD_BAL_CAT
§ MAP_CRDLN_PUR_STD_CRDLN_PUR
§ MAP_CRDLN_TYP_STD_CRDLN_TYP
§ MAP_CREDIT_SCR_MDL_REG_MDL_DDL
§ MAP_CREDIT_SCR_MDL_REG_MDL_VWI
§ MAP_DIM_IRC_STD_IRC
§ MAP_DIM_LOB_STD_LOB
§ MAP_GL_CODE_REP_LINE
§ MAP_MITG_TYP_STD_MITGN_TYP
§ MAP_PARTY_TYP_STD_PARTY_TYP
§ MAP_PROD_CODE_STD_PROD_TYPE
§ MAP_DIM_GL_ACCT_STD_GL_TYPE
§ MAP_VEHCL_TYP_STD_VEHCL_TYP
§ MAP_RECVR_TYP_STD_RECVR_TYP
§ MAP_WRTOFF_STD_WRTOFF_REASN
· Data is not populated in the target table FSI_INTRA_COMPANY_ACCOUNT.
Workaround:
FSI_REG_LEGAL_ENTITY_HIER load references POP_REG_LE_HIER DT, which is not supported in OFSDF Hive. FSI_REG_LEGAL_ENTITY_HIER must be loaded to make use of the T2T for FSI_REG_LEGAL_ENTITY_HIER because that T2T consists of an inner join on FSI_REG_LEGAL_ENTITY_HIER.
Similarly, Dimension tables for Unsupported SCDs must be loaded to make use of the T2Ts that reference them.
· In Hive, the exchange rates population for cross currencies is not being derived using exchange rates values for a base currency.
Workaround:
Source the corresponding Exchange Rate value.
· When performing Big Data installation for OFSDF Hive, the following error is logged in the file OFS_BFND_installation.log:
Error:ORA-00942: table or view does not exist
ORA-06512: at "<Atomic_Schema_Name>.FSI_CREATE_SEQUENCE", line 6 Workaround:
This error can be ignored.
This section provides the list of supported SCDs and T2Ts for OFSDF Hive (Staging and Results on Hive deployment process).
Topics:
· List of Unsupported SCDs and T2Ts
The SCDs for Hive used in Data Foundation solutions are listed in the Oracle Financial Services Data Foundation for Hive - SCD Metadata for Hive spreadsheet under Technical Metadata for OFSDF HIVE 8.1.0.0.0.
To execute FSDF_SOURCED_RUN and FSDF_EXE_RUN, use the Rules Run Framework. For more information, see the Rules Run Framework section in the Oracle Financial Services Advanced Analytical Applications Infrastructure User Guide Release 8.1.0.0.0.
The T2Ts for Hive used in Data Foundation solutions are listed in the Oracle Financial Services Data Foundation for Hive - Technical Metadata (Staging Source) spreadsheet under Technical Metadata for OFSDF HIVE 8.1.0.0.0.
This section provides the list of unsupported SCDs and T2Ts for OFSDF Hive (Staging and Results on Hive deployment process).
This is the list of unsupported SCDs with corresponding Map Reference Numbers, for OFSDF Hive (Staging and Results on Hive deployment process):
· SCD-195
· SCD-196
· SCD-205
· SCD-208
· SCD-465
· SCD-762
· SCD-761
· SCD-332
· SCD-126
· SCD-127
· SCD-128
· SCD-129
· SCD-270
· SCD-132
NOTE |
Hierarchies are supported using AM/HM screens in OFSDF (RDBMS). However, development of the underlying functionality for AM/HM feature in OFSDF Hive is planned for future releases. As a result, the tables REV_BIHIER and REV_LOCALE_HIER will be available in Hive, when the Hierarchies will be supported during future releases. |
This is the list of unsupported T2Ts for OFSDF Hive (Staging and Results on Hive deployment process):
· T2T_FCT_COMMON_CUSTOMER
· T2T_FCT_PARTY_EMPLOYMENT_DETAILS
· T2T_FCT_PARTY_ROLE_MAP
· T2T_FCT_SERV_ACCT_CREDIT_SCORE_DTL
· T2T_FCT_ACCT_CREDIT_SCORE_DETAILS
· T2T_FCT_CAP_INSTR_DETL
· T2T_FCT_SECURITIZATION_TRANCHE
· T2T_FCT_REF_CCY_SWAPS_RATE
· T2T_FCT_REF_EQ_CMDTY_VOL
· T2T_FCT_REF_EQ_DIV_RATE
· T2T_FCT_REF_FOREX_RATE
· T2T_FCT_REF_FOREX_VOL
· T2T_FCT_REF_FRA_SWAPS_RATE
· T2T_FCT_REF_INFLATION_RATE
· T2T_FCT_REF_INTEREST_RATE
· T2T_FCT_REF_MKT_CREDIT_RATE
· T2T_FCT_REF_OPTION_VOL
· fn_Pop_Reg_LE_Hier
· T2T_FSI_EXCHANGE_RATES_FRWD
· T2T_FCT_FIXED_ASSETS
· T2T_FCT_CREDIT_PARTCPN_TRNCH_DETL
· T2T_FCT_HEDGE_PORTFL_SET_ACCT_MAP
· T2T_FCT_SPEND_OBLIGATIONS
· T2T_FCT_PAYMENTS_SUMMARY
· T2T_FCT_MERCHANT_BANKING
· T2T_FCT_LOANS_SERVICED
· T2T_FCT_LITIGATION_DETAILS
· T2T_FCT_FUND_CIS_COMPOSITION
· T2T_FCT_ACCT_WRITE_OFF_DETAILS
· T2T_FCT_ACCT_RECOVERY_DETAILS
· T2T_FCT_CARDS_SUMMARY
· T2T_FDB_STG_TD_CONTRACTS
· T2T_FDB_STG_CASA
· T2T_FDB_STG_BORROWINGS
· T2T_FFSIS_STG_TRUSTS
· T2T_FFSIS_STG_MUTUAL_FUNDS
· T2T_FFSIS_STG_RETIREMENT_ACCOUNTS
· T2T_FFSIS_STG_TD_CONTRACTS
· T2T_FFSIS_STG_MANAGED_INV_ADV
· T2T_FFSIS_STG_CUSTODIAL_ACCOUNTS
· T2T_FFSIS_STG_INVESTMENTS
· T2T_FFSIS_STG_CASA
· T2T_FLAS_STG_OD_ACCOUNTS
· T2T_FLAS_STG_LOAN_CONTRACTS
· T2T_FLAS_STG_LEASES_CONTRACTS
· T2T_FCT_MORT_SERV_RIGHTS
· T2T_FCT_NET_EXPOSURES
· T2T_FCT_PARTY_SHR_HLD_PERCENT
· T2T_FSCAS_STG_CREDIT_DERIVATIVES
· T2T_FSCAS_SWAP_CONTRACTS
· T2T_FSCAS_FORWARDS
· T2T_FSCAS_FUTURES
· T2T_FSCAS_MM
· T2T_FSCAS_STG_INVESTMENTS
· T2T_FSCAS_STG_OPTIONS
NOTE |
The integration T2Ts with the OFSAA applications such as BASEL, LLFP, and LRS are not supported in Hive. |
To load data in OFSDF Hive, use the Rules Run Framework. For more information, see the Rules Run Framework section in the Oracle Financial Services Advanced Analytical Applications Infrastructure User Guide Release 8.1.0.0.0.