Set Up the Pipeline Parameters
Oracle Utilities Data Intelligence comes with pipeline parameters that apply to all functional areas. The parameters available include:
- Data Refresh Schedule: Specify the frequency and when you want the incremental data load to happen. The incremental data refresh happens once a day for each connection. While specifying the timezone, the recommendation is to use city names to handle the daylight savings. For example, instead of selecting timezone such as EST or PST, select America/Los_Angeles. In this case, the data refresh process calculates the value mentioned in the Time field based on the local time irrespective of daylight savings.
-
Initial Extract Date: Initial extract date is used as a starting point to indicate from when to extract data from the source schema when the data pipelines run. Data created after the initial extract date gets processed and loaded into to the warehouse. It can be used to control the initial data load volume.
In order to load the full data from your source application, please look into the source application and determine the earliest possible reference date from when data will be present. For example, for an expectation of data to be present in source systems (such as CCS, MSCS, WACS) from January 2000, the extract date should be 01-01-2000. This would load all the data. However, if the expectation is to load data from a later point (for example, 2010), then the date entered should be 01-01-2010.
Note:
Please choose the date carefully since this alters the load behavior of data pipelines. Once the data pipelines jobs start running, avoid changing the initial extract date. Any changes to done to his parameter would need a reset activity to be done. For details on how to do this, see About Pipeline Parameters in the Fusion Data Intelligence documentation.
Set up the pipeline parameters for your data model file before running your data pipelines for the functional areas. You can set up and schedule the parameters independently of each other.