Data Load Process
Overview of a Data Load Process
Data is the core foundation for any dashboard. The data is ingested into the Oracle Enterprise Command Center Framework engine from the source system using rules and shapes defined by the source system. The data is modeled in a way to fit various use cases addressed by the dashboard. The source of truth of the data is the Oracle E-Business Suite system. Oracle Enterprise Command Center Framework architecture relies on the Oracle E-Business Suite database to run the data load process from Oracle E-Business Suite into Oracle Enterprise Command Center Framework. It does not introduce new complex tools or technologies but rather relies on the power and flexibility of SQL and PL/SQL to prepare and retrieve the data in the shape required to power the dashboards.
The following diagram shows the high-level architecture of a data load program. An administrator can use concurrent programs to trigger data load execution in Oracle Enterprise Command Center Framework to pull Oracle E-Business Suite data into ECC data sets. This ingestion of data lets can be done in a full data load or incremental data loads. A business user benefits from the data ingested into Oracle Enterprise Command Center Framework through the Enterprise Command Center. The Enterprise Command Center sends requests for pages that read data from the ECC data sets, and these pages are then returned to the Enterprise Command Center.
High-Level Architecture of Data Load Programs
Oracle Enterprise Command Center Framework supports online data load for minimizing the impact of downtime. Users can still use the dashboard while data load runs in the background. This feature works with the dual file system: an active/query data set for serving the queries, another data set for the extract, transform, and load (ETL) process. Once the full load process is completed, the query data set is deleted and ETL data set replaces the active data set. For details on enabling online full load, refer to Enabling Online Full Load.
Oracle Enterprise Command Center Framework supports sequential full load. This feature allows for a multi-language full load to be executed for one particular language per execution rather that for all configured languages at once. This approach provides better control on overhead on the database side and can result in better overall performance and optimal ETL query executions. For details on enabling sequential full load, refer to Enabling Sequential Full Load.
Data Load Process Definitions
Full Load
When you first install Oracle Enterprise Command Center Framework and import an application for a particular business area, one of the first steps you do is run full data load. The process creates the relevant data set(s) inside the Oracle Enterprise Command Center Framework engine and starts the ingest process from the Oracle E-Business Suite database into Oracle Enterprise Command Center Framework. Full load also provides an opportunity to recreate the data set and repopulate it with fresh data if the data set has already been created.
A full load for a business dashboard is triggered by a concurrent job owned by the same business area. The program triggers the associated data set load rules defined in Oracle Enterprise Command Center Framework. Every data set can have more than one rule to load data with each of those containing one or more queries against the Oracle E-Business Suite database. Oracle Enterprise Command Center Framework connects to Oracle E-Business Suite using the EBS SDK to allow the data load code to run any data preparation, cleansing and wrangling logic before data is ingested into Oracle Enterprise Command Center Framework.
Incremental Load
Incremental data load follows the same logic and procedure as that for data load with two main differences. First, incremental load expects that the data set is already created and its metadata be already captured and will fail if these conditions are not met. Second, incremental data load will pick up only the modified data from Oracle E-Business Suite as of the time of the last successful full or incremental load.
File Upload
Oracle Enterprise Command Center Framework allows ingesting data into the data set using a CSV file. File upload provides the flexibility to use a file with any custom text separator and is specific to the language of the data in the file. File upload can be used to completely refresh the data or just extend existing data in data set. For more information, refer to Ingesting Data, Oracle E-Business Suite: Administering Enterprise Command Centers.
Query Upload
Oracle Enterprise Command Center Framework allows ingesting data into the data set using an SQL query on any connected database. Query Upload can also completely refresh the data or extend existing data in data set. For more information, refer to Ingesting Data, Oracle E-Business Suite: Administering Enterprise Command Centers.