About Data Refresh Performance

Oracle strives constantly to improve performance for data loading in pipelines.

The performance of loading data for your instance will vary. The time to complete data processing, both full warehouse loads and incremental data loads, depends on various factors. A data pipeline load includes the following:

• Extracting data from the Oracle Fusion Cloud Applications.
• Loading the data into Oracle Autonomous Data Warehouse.
• Transforming the data into the prebuilt schema.

The time to complete each of these steps is determined by various factors including:

• The availability of the source system.
• The size and complexity of the source data.
• The activated functional areas.
• Custom SQL queries that impact Oracle Autonomous Data Warehouse.
• Your queries running concurrently.
• Customizations made on the source system objects (which require a full load for those objects).
• The patching of source and target systems.

For daily refresh of data and pipeline execution to update all the resulting key metrics and dashboards, the execution time depends on many factors such as the amount of data getting incrementally refreshed. In order to plan your downstream workflow needs, ensure to perform these actions:

• Turn on the pipeline execution estimation using the Data Refresh Estimate preview feature on the Enable Features page. The estimated refresh completion details are visible on the Pipeline Settings page.
• Trigger any subsequent operations based on the estimated refresh completion time.
• Reach out to Oracle Support if the refresh isn't completed by the estimated completion time.
• Subscribe to Event Notifications, so you can receive an email notification as soon as there is any issue with the pipeline execution. See Enable Event Notifications (Preview).

Data refresh is typically completed daily unless the data pipeline jobs are turned off or stuck. You can observe data loading times for your specific source and warehouse configuration to estimate the time it takes for an incremental daily refresh and for a full warehouse load. This information can help you plan for the optimal time in the day to start your daily data refresh. You may want to schedule the data load to run during off-peak hours, for example, run initial full warehouse loads during weekends and incremental loads during weeknights to ensure that users aren't impacted.

You can view the estimated refresh completion time for daily pipelines on the Pipeline Settings page in the Estimated Refresh Completion field as a Preview feature. This enables you to plan your tasks in the application.

The incremental data refresh process refreshes objects in multiple stages to ensure essential data is made available quickly. The multiple stage refresh process includes:

• A priority refresh of specific tables that you configured for operational use cases.
• Next, the process refreshes the prebuilt modules and custom data configurations where applicable. The objects refreshed in this stage are part of the core pipeline and sourced directly from Oracle Fusion Cloud Applications and needed for core content (for example, transactions and dimensions).
• Lastly the process refreshes the data augmentations of type Dataset, data augmentations configured as Versioned datasets, data augmentations with no Last Modified Date or Last Update Date, and the prediction modules such as Supplier On-time Payments Prediction. The objects refreshed in this stage require additional processing; for example, datasets used in prebuilt machine learning use cases).

The system predicts and displays the incremental load completion time. If you haven't activated content that is processed last in the multi-stage refresh process, you won't see this information on the Pipeline Settings page in the Data Pipeline section.