This section provides information about Big Data processing supported in the Oracle Insurance Data Foundation application.
Topics:
OIDF supports Big Data processing on HDFS (Hadoop Distributed File System) using Hive as Query Engine. The AAI application components such as SCD, H2H, and DQs provide equivalent Hive support. For more information, see OFS Analytical Applications Infrastructure User Guide.
In Big Data processing, OIDF provides two types of deployment processes for Hive support. They are as follows:
· Staging on Hive and Results on RDBMS
This section consists of the following topics:
· List of Supported SCDs and T2Ts
· Executing Run through Rule Run Framework for Hive
In the Staging and Results on Hive deployment process, the Staging, and Results model elements are a part of Hive known as Hive Datadom. The config schema definitions used by AAI and certain metadata definitions that were part of Atomic schema now reside in an RDBMS schema known as Metadom. See the following architecture diagram for the representation of this deployment process.
Figure 141: Data Foundation Big Data Architecture with Staging and Results on Hive
For the list of supported OIDF Hive Metadata definitions, see the Run Chart, SCD Metadata, and Technical Metadata (Staging Source) documents at My Oracle Support (MOS).
NOTE:
Ensure that String values, which are null, must contain \n and then source to Hive stage tables.
In the Staging on Hive and Results on the RDBMS deployment process, the Staging occurs on Hive, known as Hive Datadom, and the Results model elements are a part of RDBMS, known as RDBMS Datadom. In this deployment process, additional software is required, which is Oracle Big Data SQL (For the software version information, see the Oracle Financial Services Analytical Applications Technology Matrix Release 8.1.0.0.0). Oracle Big Data SQL software resides on Hive and RDBMS Datadoms and enables the user to create a link to Hive Stage tables as external tables in RDBMS. This software manages the data representation of the Hive tables in RDBMS Datadom and can be accessed as external table objects. As a result, enables the use of RDBMS infrastructure components of SCD, T2T, and DQs. See the following architecture diagram for the representation of this deployment process.
Figure 142: Data Foundation Big Data Architecture with Staging on Hive and Results on RDBMS
Modifications in OIDF Hive (Staging and Results on Hive deployment process) compared to OIDF RDBMS are:
· Dim Dates Population
The Dim Dates functionality in Hive is similar to that of RDBMS. The difference exists in the date input format on the UI. In OIDF Hive, the Date Range input format is, an executable textbox contains .sh file name followed by the FROM and TO dates.
For example:
load-dimdates-run.sh,20110101,20110105
The list of workarounds in OIDF Hive (Staging and Results on Hive deployment process) is:
Workarounds:
§ Data Redaction
The process of enabling Data Redaction for Hive can be performed using CDH Manager (Cloudera Distribution Hadoop Manager). For more information, see the section Sensitive Data Redaction in the Cloudera Security Guide.
§ Right to Forget
The Right to Forget feature enhancement is planned for future releases. However, prior to loading in the Hive instance, Anonymization of Party PII data can be done at the source. This process provides the flexibility to secure the PII information, and in addition, to drop the PII upon a request from the Party.
· Mappers are supported using AMHM screens in OIDF (RDBMS). However, the development of the underlying functionality for the AMHM feature in the OIDF Hive is planned for future releases. Therefore, the Mappers will be supported during future releases.
Workaround:
The Views defined on these Mappers in RDBMS are converted to Tables in Hive. Therefore, these Mappers must be loaded manually:
§ MAP_BAL_CAT_STD_BAL_CAT
§ MAP_CRDLN_PUR_STD_CRDLN_PUR
§ MAP_CRDLN_TYP_STD_CRDLN_TYP
§ MAP_CREDIT_SCR_MDL_REG_MDL
§ MAP_DIM_GL_ACCT_STD_GL_TYPE
§ MAP_DIM_IRC_STD_IRC
§ MAP_DIM_LOB_STD_LOB
§ MAP_GL_CODE_REP_LINE
§ MAP_MITG_TYP_STD_MITGN_TYP
§ MAP_PARTY_TYP_STD_PARTY_TYP
§ MAP_PROD_CODE_STD_PROD_TYPE
§ MAP_RECVR_TYP_STD_RECVR_TYP
§ MAP_VEHCL_TYP_STD_VEHCL_TYP
§ MAP_WRTOFF_STD_WRTOFF_REASN
NOTE:
Hierarchies are supported using AMHM screens in OIDF (RDBMS). However, the development of the underlying functionality for the AMHM feature in the OIDF Hive is planned for future releases. As a result, the tables REV_BIHIER and REV_LOCALE_HIER will be available in Hive, when the Hierarchies will be supported during future releases.
· Data is not populated in the target table FSI_INTRA_COMPANY_ACCOUNT.
Workaround:
FSI_REG_LEGAL_ENTITY_HIER load references POP_REG_LE_HIER DT, which is not supported in OIDF Hive. FSI_REG_LEGAL_ENTITY_HIER must be loaded to make use of the T2T for FSI_REG_LEGAL_ENTITY_HIER because that T2T consists of an inner join on FSI_REG_LEGAL_ENTITY_HIER.
Similarly, Dimension tables for Unsupported SCDs must be loaded to make use of the T2Ts that reference them.
· In Hive, the exchange rates population for cross currencies is not being derived using exchange rates values for base currency.
Workaround:
Source the corresponding Exchange Rate value.
· When performing Big Data installation for OIDF Hive, the following error is logged in the file OIDF_installation.log:
Error:ORA-00942: table or view does not exist
ORA-06512: at "<Atomic_Schema_Name>.FSI_CREATE_SEQUENCE", line 6
Workaround:
This error can be ignored.
This section provides the list of supported SCDs and T2Ts for OIDF Hive (Staging and Results on Hive deployment process).
The SCDs for Hive used in Data Foundation solutions are listed in the Oracle Insurance Data Foundation for Hive - SCD Metadata for Hive spreadsheet under Technical Metadata for OIDF HIVE 8.1.1.0.0.
Deploy the OIDF Hive T2Ts using the Rules Run Framework. For more information, see the Rules Run Framework section in the Oracle Financial Services Advanced Analytical Applications Infrastructure User Guide Release 8.1.1.0.0.
The T2Ts for Hive used in Data Foundation solutions are listed in the Oracle Insurance Data Foundation for Hive - Technical Metadata (Staging Source) spreadsheet under Technical Metadata for OIDF HIVE 8.1.1.0.0.
· SCD-195
· SCD-196
· SCD-205
· SCD-208
· SCD-465
· SCD-411
· SCD-236
· SCD-237
· SCD-242
· SCD-248
· SCD-244
· SCD-336
· SCD-337
· SCD-378
· SCD-381
· SCD-335
· SCD-409
· SCD-419
· SCD-460
· SCD-332
· SCD-132
This is the list of unsupported T2Ts for OIDF Hive (Staging and Results on Hive deployment process):
· T2T_FSI_GL_CODE_REP_LINE_HIST
· T2T_FCT_PARTY_EMPLOYMENT_DETAILS
· T2T_FCT_PARTY_EXAM_DETAILS
· T2T_FCT_PARTY_PD_DETAILS
· T2T_FCT_PRDCR_INS_DISTRBTION_DTLS
· T2T_FCT_PARTY_MDCAL_CONDITN_DTL
· T2T_FCT_PARTY_FAMILY_MEDICAL_DA
· T2T_FCT_COMMON_CUSTOMER
· T2T_FCT_APPLICATION_DOCUMENT
· T2T_FCT_APPLICNS_DOC_PRINT_LOG
· T2T_FCT_PARTY_INS_POLICY_ROLE_MAP
· T2T_FCT_VEHICLE_INSPECTION_DETAILS
· T2T_FCT_VEHICLE_DEVICE_EVENT_DTLS
· T2T_FCT_VEHICLE_FLEET_MAP
· T2T_FCT_ACCT_CREDIT_SCORE_DETAILS
· T2T_FCT_CLAIM_BREAK_UP_DETAILS
· T2T_FPT_STG_ANNUITY_TXNS
· T2T_FPT_STG_HEALTH_INS_TXNS
· T2T_FPT_STG_LIFE_INS_POLICY
· T2T_FPT_STG_PROP_CASU_POLICY_TXNS
· T2T_FPT_STG_RETIREMENT_ACCOUNTS_TXNS
· T2T_FCT_POLICY_TRANSACTIONS_HLD
· T2T_FCT_POLICY_TRANSACTIONS_ISS
· T2T_FCT_INSURANCE_MORTALITY
· FN_POP_REG_LE_HEIR
· T2T_FSI_EXCHANGE_RATES
· T2T_FSI_EXCHANGE_RATES_FRWD
· T2T_FCT_PRODUCT_RATE_MATRIX
· T2T_FCT_TRIP_DETAILS
· T2T_FCT_TRIP_FLEET_SUMMARY
· T2T_FCT_TRIP_SUMMARY
· T2T_FCT_DRIVER_BEHAVIOUR_SUMMARY
· T2T_FCT_VEH_PERFORMANCE_SUMMARY
· T2T_FCT_INTRA_COMPANY_ACCT_SU
· T2T_FSI_INTRA_COMPANY_POLICY
· T2T_FCT_FIXED_ASSETS
· T2T_FCT_HEDGE_PORTFL_SET_ACCT_MAP
· T2T_FLAS_STG_LOAN_CONTRACTS
· T2T_FCT_FUND_CIS_COMPOSITION
· T2T_FFSIS_STG_INVESTMENTS
· T2T_FFSIS_STG_MANAGED_INV_ADV
· T2T_FFSIS_STG_MUTUAL_FUNDS
· T2T_FFSIS_STG_TRUSTS
· T2T_FFSIS_STG_CUSTODIAL_ACCOUNTS
· T2T_FCT_LITIGATION_DETAILS
· T2T_FCT_PAYMENTS_SUMMARY
· T2T_FCT_ACCT_RECOVERY_DETAILS
· T2T_FCT_ACCT_WRITE_OFF_DETAILS
· T2T_FCT_SPEND_OBLIGATIONS
· T2T_FCT_CREDIT_PARTCPN_TRNCH_DETL
Deploy the OIDF Hive T2Ts using the Rules Run Framework. For more information, see the Rules Run Framework section in the Oracle Financial Services Advanced Analytical Applications Infrastructure User Guide Release 8.1.1.0.0.