Machine Learning Services Overview

Note: Use of an external service to connect to IoT IA services has been deprecated and will be removed in a future release. IoT IA is being replaced by Embedded Machine Learning.

Predicting shipment transit times can be a challenge in today's ever changing environment. Logistics Machine Learning (LML) allows you to use historical data to predict end-to-end transit times for direct shipments. LML leverages OTM historical shipment and order data with optimized algorithms and machine learning techniques to deliver highly accurate predictions.

Two platforms can be used to predict transit times and order routes depending on how OTM is configured:

• External Service (for use only with IoT IA services and can be used by migrated instances only. This is being deprecated.).
• Embedded Learning (see Embedded Machine Learning Overview for mere details).

The Logistics Machine Learning transit time estimation process is a continuous processes in which the information gained from the results of machine learning training is used to both make changes to current process and to update setup data that can then be used to enhance machine learning model training.

Let's review the high-level flow:

1. Data Setup: Machine learning needs data as input. LML provides historical OTM data via machine learning projects and machine learning scenarios. This data can include shipment and tracking event data.
2. Data Export and Pre-Training Analytics: Export this historical data to Oracle Transportation Intelligence (OTI) and review it to see if you want to change the data setup.
3. Training and Post-Training Analytics: Send machine learning scenario data to Oracle's IoT IA machine learning service for training. Then, review the learning results.
4. Prediction: Select groups of OTM shipments and obtain a predicted transit time for each shipment from the IoT IA machine learning service or embedded machine learning.
5. Evaluation: Finally, you evaluate all of this data. Set up OTM process management to automate the above steps.

IoT IA (Internet of Things Intelligence Applications) Process Flow

Transit Time Flow

Let's take a deeper look at the process and see how the shipment related data moves between OTM, OTI, and IoT IA.

1. Set up, Data Extraction (Export), and Loading
   
   This flow involves:
   1. In Logistics Machine Learning, you set up your machine learning project. The project collects historical shipment and tracking event data via shipment saved queries.
   2. Still in LML, you export the data to Logistics Machine Learning Intelligence in OTI. In this step, LML extracts the data that the project collects and loads it into OTI HDOWNER database.
   3. In OTI, you review the data in either the Logistics Machine Learning intelligence subject area or the dashboard, correct the data as necessary, and repeat step b to export the updated data to OTI.
   4. Back in LML, you send the data to IoT IA using the Export to Machine Learning Services action. This action saves the data sent to Analytics to a flat file, copies the flat file to the object store, and tells the IoT IA machine learning service to pick up the file. When IoT IA machine learning service picks up the file, it sends a response to LML.
2. Train a Scenario
   
   Now that your data is uploaded to the IoT IA server, you are ready to filter that data into smaller sub-sets, and tell IoT to train the machine learning model using these smaller sub-sets of the data.
   1. LML: Use a machine learning scenario to filter the data using shipment filters, exclude columns, and remove outliers.
   2. LML: Use the Load Shipment Data into Analytics action to send information to Analytics about the sub-set of shipments that will be used in training for a given scenario.
   3. LML to IoT IA: Perform training on these sub-sets of data via the LML action Perform Training. Training is performed on the IoT IA server. Once the model is trained, you can review the results of the training.
3. Predict Transit Times
   
   And finally, you are ready to use the trained model to predict shipment transit times using the Perform Prediction action.
4. System Log IDs are available to help troubleshoot LML issues.

Enabling Logistics Machine Learning

Logistics Machine Learning involves several different Oracle systems which need to communicate with each other. OTM, OTI, OCI Object Store, and IoT IA are all included in the LML data flow. See the Setup Guide for Logistics Machine Learning for a detailed look at setting up Logistics Machine Learning.

Enabling Transit Time Machine Learning

Oracle Transportation Intelligence (OTI)

1. Ensure that OTI is installed and the Run ETL action has been performed.
2. Provide access to Logistics Machine Learning Intelligence subject area and dashboard.

IoT IA

1. Subscribe to IoT IA.
2. Complete the steps to integrate with the IoT Fleet Monitoring Cloud Service.
3. Enable Transit Time Predictions for Shipments.
4. Create an external system with the following:
   1. User Name and Password as provided by IoT IA
   2. Content Type: application/json
   3. Authentication Type: HTTP Authentication (Basic)
   4. URL as provided by IoT IA
5. Attach the external system to the Machine Learning Logic Configuration via the ML EXTERNAL SYSTEM ID parameter.

Oracle Storage Cloud Service

Along with your subscription to the Oracle Storage Cloud Service, you can set up a pre-authenticated request and get a pre-authenticated URL. Then, create an external system that uses this pre-authenticated URL.

Machine Learning Project

This page is accessed via Logistics Machine Learning > Machine Learning Project.

The Logistics Machine Learning (LML) project defines the scope of the historical data with which to train the machine learning model. A machine learning project can consist of multiple scenarios each representing a different business segment. There are two options you can use to train historical shipment data: machine learning project to consider all North American shipments. Then use machine learning scenarios to narrow down the data to consider shipments with different modes such as truckload, LTL, and parcel shipments.

The Machine Learning Project screen consists of machine learning project details, project data, and a grid with all the machine learning scenarios associated with the project. A machine learning project can consist of multiple scenarios each containing a different sub-set of order releases or shipments. Then, you can compare training data for all of the scenarios.

Adding a Machine Learning Project

1. Enter a Project ID field. Add a meaningful ID for your project. There is a strong possibility that you will generate many projects with slightly different shipment or order release data. So having a good way to identify and differentiate between your many projects based on the ID will become very beneficial. For example, ETA_NA.
2. Enter a Project Name. Add a descriptive name for your project. For example ESTIMATED TRANSIT TIME FOR NA.
3. Enter a Description for this project that captures the purpose of this project and the scenarios involved.
4. Select a Domain Name.
5. Select an Objective of "External Service". You can select one of the following Objective Model Types:
   1. Planned ETA Prediction: This objective model type uses shipment-related attributes as the input to the model at the planning time or when the actual departure time at the source is known. This objective model type provides a base prediction before any in-transit tracking events are received.
   2. Planned and Event based ETA Prediction: This objective model type uses planned ETA prediction (as described above) and adds in-transit tracking event information to further refine the predicted ETA.
6. When the Objective is External Service, you can optionally enter a Logic Configuration ID.
7. Optionally, add a New Machine Learning Scenario.

The Last Exported to Machine Learning field is populated when you run the Export to Machine Learning Service action. You must click Refresh in the Export to Machine Learning Services results page.

Adding Saved Queries

On the Project Data tab, you can add saved queries to pull in the required historical shipment data.

1. Select a Saved Query Type:
   • If Planned ETA Prediction: This objective model type uses shipment-related attributes as the input to the model at the planning time or when the actual departure time at the source is known. This objective model type provides a base prediction before any in-transit tracking events are received. This objective model type is available if you selected an Objective of External Service or Embedded Learning.
   • Planned and Event based ETA Prediction: This objective model type uses planned ETA prediction (as described above) and adds in-transit tracking event information to further refine the predicted ETA. This objective model type is available if you selected an Objective of External Service.
2. When the Objective is External Service, you can optionally enter a Logic Configuration ID.
3. Enter one or more Saved Query IDs and click Save to populate the grid.

Deleting a Machine Learning Project

The following scenarios can occur when deleting a machine learning project:

• If the project and scenarios are only in OTM (no actions have been run), the project and all scenario data associated to the project is deleted.
• If the project data exists in both OTM and OTI (the Load Data Analytics action), the project and all scenario data associated to the project is deleted. Also, the corresponding data in Logistics Machine Learning Intelligence subject area is deleted.
• If the project data exists in OTM, OTI (using the Load Data Analytics action), and IoT (using the Export to Machine Learning Services and Perform Training actions), the data is deleted from IoT first. Only when the deletion of the IoT data is successful is the project and all scenario data associated to the project is deleted. Also, the corresponding data in Logistics Machine Learning Intelligence subject area is deleted. 
  
  If the deletion of the IoT data fails, then nothing is deleted and you see an error message.

Machine Learning Scenario

This page is accessed via Logistics Machine Learning > Machine Learning Scenario.

A scenario defines one or more particular use cases on which you want to run machine learning algorithms. For example, consider shipments with a particular transport mode, such as truckload, LTL, and parcel.

You can include scenarios as part of a machine learning project by clicking New Machine Learning Scenario on the Machine Learning Project screen. This allows you to run training for all of the scenarios on the project and compare the results. For example, you can include three scenarios: one for truckload, one for LTL, and one for parcel transport modes. Then, you can compare the results.

Machine learning scenarios allow you to use the following to limit your historical data:

• Scenario filter allows you to select a different set of shipments or orders to consider in each of your scenarios.
• Scenario Outlier filter allows you to select a different set of shipments or orders and identify outliers. For shipments, this is based based on transit time to be excluded from training.
• Excluded columns allows you to exclude attribute columns.

Adding a Machine Learning Scenario

1. Enter a meaningful Scenario ID.
2. Enter a Scenario Name such as ETA_NA.USA_TL.
3. Enter a Description to capture the purpose of the scenario.
4. Select a Domain Name.
5. Enter the Project ID if this is created directly. If creating the modeling scenario from a modeling project, the project ID is entered automatically.
6. Select an Objective Model Type depending on what is on the machine learning project:
   • an Objective Model Type of Planned ETA Prediction, you see a single option of Planned ETA Prediction. This option creates machine learning models that only consider shipment-level histories and are good for predicting shipments that are either not executed or do not have any tracking events.
   • an Objective Model Type of Planned and Event based ETA Prediction, you see two options: Event Based ETA Prediction and Planned ETA Prediction.
     
     Event Based ETA Prediction creates machine learning models that not only considers shipment-level information but also tracking events. These models are good for predicting shipments that are in transit.
7. Create filters.
   • Scenario Filter
   • Scenario Outlier Filter
8. Add Excluded Columns.

Scenario Filters

Use scenario filters to narrow down the list of shipments/orders based on shipment/order attributes. For example,

• only include shipments with the transport mode of TL
• only include shipments with a transport mode of TL, a source location country of USA, and a destination location country also of USA.

To create a new scenario filter:

1. In the Scenario Filters grid, click New Filter.
2. Enter a Filter Name
3. Select a Column Name.
4. Enter or select a Column Value and an operator. Not all operators are supported. The allowed operators are dependent on the Column Name.
5. Click Save.
6. Enter any additional shipment filter criteria as needed. Click Save after each.
   
   If you define multiple scenario filter criteria, records satisfying all of the criteria would be included in the result set.

Scenario Outlier Filters

Outlier filters allow you to specify actual transit time constraints to limit shipments included in the scenario data. For example, you can add an outlier filter that excludes shipments where the transit time is less than 1 day and 12 hours or more than 10 days. Scenario outlier filters are only available for the Objective Model Types of Planned ETA Prediction and Planned and Event based ETA Prediction.

1. In the Outlier Filters grid, click New Filter.
2. Enter a Filter Name.
3. Optionally, enter Filter Criteria:
   1. Select a Column Name.
   2. Enter or select a Column Value and an operator. Not all operators are supported. The allowed operators are dependent on the Column Name.
   3. Click Save.
4. In the Outlier Criteria section, notice the Target Column with a hard-coded value of Actual Transit Time. You can only enter less than values and greater than values for the target of Actual Transit Time.
5. Enter one or both of the outlier bounds.
   • Enter a Less Than Value.
   • Enter a Greater Than Value.

Included and Excluded Columns

You can select certain columns and have them excluded from the scenario data. If excluded, a column cannot become a feature in the training model created for the scenario. Columns containing data required for training cannot be excluded and will not appear in the list.

Based on what is selected in the Objective Model Type field on the project, this section changes as follows:

• If the Objective Model Type is Planned ETA Prediction, you see a Planned ETA Prediction Model section. If you expand Shipment Columns you see all of the included columns and all flex field columns are excluded.
• If the Objective Model Type is Planned and Event based ETA Prediction, you see both a Planned ETA Prediction Model section (as explained above) and an Event based ETA Prediction Model section. For Event based ETA Prediction Model, you also see default included and excluded columns (which include some shipment columns and flex field columns).

Using the included and excluded column lists:

• To add columns to the Excluded Columns list, select one or more entries in the Included Columns list and click > or >> to move them to the Excluded Columns list.
• To remove columns from the Excluded Columns list, select one or more entries in the Excluded Columns list and click < or << to move them to the Included Columns list.

Deleting a Machine Learning Scenario

If you delete the project, all scenario data associated to the project is deleted. Also, the corresponding data in Logistics Machine Learning Intelligence is deleted.

Perform Training

This page is accessed via:

• Logistics Machine Learning > Machine Learning Project > Actions > Perform Training
• Logistics Machine Learning > Machine Learning Scenario > Actions > Perform Training

Note: The actions Load Data into Analytics and Export to Machine Learning Service must be completed before running this action. 

Note: This action is only available for DBA.ADMIN and .ADMIN users.

The Perform Training action tells the machine learning service to start a training run based on the data from the scenario.

You can perform training from either a machine learning project or a machine learning scenario. If you perform training from a machine learning project, you see a page where you select multiple scenarios to run the training for. You can also run Perform Training as a process via process management.

Once you run the action, you see the Feature Importance Summary page:

• Project ID: This is the machine learning project ID associated to the scenario data used for training.
• Scenario ID: This is the ID of the machine learning scenario used for training.
• Training Status: Values are:
  • TRAINING NOT INITIATED
  • INITIATED
  • ACCEPTED: The machine learning service accepted the request to train machine learning scenario.
  • PROCESSING: The machine learning service is currently training the scenario.
  • COMPLETED: The training successfully completed.
  • FAILED: The training failed.
• Model Accuracy: Contain values sent to OTM from the machine learning service or the embedded learning server.
• High Confidence Value and Low Confidence Value: Contain values sent to OTM from the machine learning service once the training completes.

Click Refresh to view the results of the training.

When the training is completed, the page updates to include a Feature Importance Summary grid which contains the following information:

• Project ID
• Scenario ID
• Feature: Significant characteristics of shipments that impact transit time when training a model.
• Percentage Contribution: Importance of the corresponding feature to the model.

If any of your scenarios fail to train, you see a Failure Reasons grid.

Perform Prediction

This page is accessed via Buy Shipment > Actions > Logistics Machine Learning > Perform Prediction

Note: The actions Load Data into Analytics, Export to Machine Learning Service, and Perform Training must be completed before running this action. 

Prerequisites: For a Platform/Objective of External Service, several properties must be set for this action to work when using Machine Learning Services and IoT. They include the URL of the Oracle Machine Learning Service and log in details.

The Perform Prediction action sends either sends all buy shipment information or order release information to the machine  attributes and event attributes along with the necessary scenario information to the machine learning service. For ETA Prediction based on the request, the machine learning service (IoT A) provides the Predicted Transit Time for the selected shipment. The action populates the Predicted Arrival Time on the last Dropoff stop of the shipment. Once the prediction data is received from the machine learning service, this action populates the analytics prediction tables in Transportation Intelligence.

1. Select one or more buy shipments.
2. Select Actions > Logistics Machine Learning > Perform Prediction. The Perform Prediction action opens the Prediction Input page where the following input is required:
   1. Enter a machine learning Project ID.
   2. Select a Scenario Model Type.
      • If the machine learning project has an Objective Model Type of Planned ETA Prediction, you see a single option of Planned Model.
      • If the machine learning project has an Objective Model Type of Planned and Event based ETA Prediction, you see the following options: Planned Model, Event Model, or Both.
   3. Select a Scenario ID from the resulting list. This list is empty if the specified project ID has no scenarios for which the Perform Training action was completed.
   4. Click Perform Prediction.

When the prediction is in process, you see the Prediction Results page that may include the following information in the Prediction Status section:

• Project ID
• Scenario ID
• Status: ACCEPTED, PROCESSING, COMPLETED, or FAILED

When the prediction is complete, a Shipment Predictions section is added to the page and it contains:

• Shipment ID
• Event ID (depending on the Scenario Object Model Type selected): This is the tracking event ID.
• Accuracy: This is the accuracy of the model used in the prediction. Accuracy is displayed in decimals.
• Predictions: This column lists the predicted transit time of the shipment. Prediction interval provides a range of values that predicts the value of a new observation, based on your existing model. A 95% prediction interval of 9H 30M and 10H 30M tells you that future transit time for shipments with similar attributes will fall into that range 95% of the time. There is a 5% chance that a transit time will not fall into this interval.
• Prediction Low Value: Indicates the lower end of the prediction interval.
• Prediction High Value: Indicates the higher end of the prediction interval.

If the prediction failed, you see a Failure Reasons section with details of what went wrong.

Compare Prediction Results

This page is accessed via Logistics Machine Learning > Machine Learning Results > Shipment Prediction Results. Search for and select prediction results and click Compare Prediction Results

Note: The actions Load Data into Analytics, Export to Machine Learning Service, and Perform Training must be completed before running this action.

Prerequisites: Several properties must be set for this action to work. They include the URL of the Oracle Machine Learning Cloud Service and log in details.

Shipment Prediction Results

The action shows you the machine learning service predictions for all of the selected shipments so you can compare them side-by-side.

When you run this action, the Shipment Prediction page opens and may include the following information:

• Project ID
• Scenario ID
• Shipment ID
• Event ID (depending on the Scenario Objective Model Type selected on the Perform Prediction input screen): This is the tracking event ID.
• Status: ACCEPTED, PROCESSING, COMPLETED, or FAILED
• Model Accuracy: This is the accuracy of the prediction displayed in a percentage.
• Predictions: This column lists the predicted transit time (09H 00M 00S) of the shipment.
• Prediction Low Value: This field displays the ideal transit time (08H 30M 00S) if there are no shipment delays.
• Prediction High Value: This field displays the transit time (09H 30M 00S) if there are no shipment delays.

Click the arrow to view more shipment details, such as the planned, predicted, and actual values for the following:

• Start Time
• End Time
• Transit Time, and
• Deviation (between the planned and predicted values versus the shipment actuals).