Run ETL

This page is accessed via:

• Transportation Intelligence > Process Management > Run ETL
• Global Trade Intelligence > Process Management > Run ETL

Note: You must be logged in as DBA.ADMIN to access this page.

Run ETL runs the ETL from the replicated operational database (ROD) or the online transaction processing (OLTP) system to the historic database (HD). This page allows you to schedule the ETL to load the data.

The first time you run ETL all of your data will load. All of the other times that you run the ETL the data load is incremental.

Note: This process does not load target data. To load target data use the Run Targets process.

Note: Never run ETL processes at the same time. It is recommended that you first start the Run ETL process and once that completes then you can run the Run Targets or Load User Information for Analytics processes.

Note: To see complete logging information like ODI Log Level, ODI User, ODI Scenario name, etc., turn on both the Workflow and ETL log IDs.

There are properties that must be set for this process to run. See the topic glog.odi Properties files. Depending on the following properties, the run ETL will either load either the Global Trade Intelligence (GTI) or Transportation Intelligence (TI) tables or both:

• ALLOW_ADVANCED_ANALYTICS is the property that controls if TI is running. If this is set to TRUE, ETL loads TI tables only.
• isAllowedGTIAnalytics is the property that controls if GTI is running. If this is set to TRUE, ETL loads GTI tables only.
• if both properties are set to TRUE, then ETL loads both TI and GTI tables.

Running the ETL

1. The Last Refresh Date displays the date the last time that Run ETL was run. The TI/GTI database connection properties need to properly setup to get this information. This will be blank if the data has not yet been refreshed. It will also be blank if the ROD is not available. In that case, check the availability of the ROD.
2. Setup the email information if you want an email with an attachment sent when the process is completed. The attachment contains all the rows from the e_load_log table for this run. For example, you may want to schedule the process to run at 2 a.m. every day, and send an email. Then someone can review the email in the morning and confirm that the process ran okay.
   • • Enter an E-mail Subject, for example, "TI/GTI Daily Scheduled ETL".
     • Enter the E-mail Message, for example, "Results are attached".
     • Enter the E-mail To(Contact ID). This is selected from the Contact IDs e-mail address chosen as To Address.
3. Specify a Lag Time Minutes (ROD Refresh Start to ETL Start) that is longer than the duration of the ROD refresh start to the ETL start.
   
   If using an ROD, there is a risk that the ETL will miss data. This can happen if the change in Oracle Transportation Management occurs between the start of the ROD refresh and the start of the ETL.
   
   To avoid this risk, specify a Lag Time Minutes (ROD Refresh Start to ETL Start) that is longer than the duration of the ROD refresh start to the ETL start. Transportation Intelligence/Global Trade Intelligence will include ROD records changed since the previous ETL but will also go further back in time based on the lag time you specify.
   
   If using Oracle GoldenGate, the Lag Time Minutes (ROD Refresh Start to ETL Start) can be as small as a minute but should be larger than 0. If you do not use a ROD, you can set the Lag Time Minutes (ROD Refresh Start to ETL Start) to 0.
   
   The default Lag Time Minutes (ROD Refresh Start to ETL Start) is 0 minutes.
4. Select the Lookback ETL check box to load lookback ETL then set the Lookback Date. The lookback functionality allows you to load data for a date other than today.
5. If Lookback ETL is selected, select a Lookback Date. For example, select a date of January 1 2020 and the ETL loads data for January 1 to January 14 (no matter what today's date is).
6. If you are running cluster scalability, specify a cluster to which this request should be assigned.
7. Enter a User Role. When you run, publish, or schedule a recurring process, you can specify the user role used to perform the process, just like when you run agents. The user role can be used to specify a separate VPD role or another domain to run the agent. When a recurring process is modified, the user role of the recurring process can be modified as well.
8. Select a Log Profile to generate focused logging when reproducing an error for the specific action. You cannot select a log profile when you choose to schedule a process, as automatic generation of action logs is not supported.
9. Select Execute, Publish, or Schedule to determine when the process will be executed.
10. Click Submit.