Use Automated Hyperparameter Tuning to Improve Forecast Accuracy

Improve the forecast accuracy of Oracle Demand Management machine learning models by using your enterprise's historical demand data to tune engine parameters to their best values. By using this approach, you’ll reduce the need to engage with forecasting model experts to tune the models. The feature introduced in this update runs in the background at individual forecasting levels, for example, an item and organization combination. The process then determines the best settings for the forecasting parameters appropriate for that combination. You can review the forecast and predicted accuracy of the suggested settings and compare the predicted results to the existing values. You can then use the suggested settings the next time you run the plan. The background process can be run for the entire forecasting data set or a subset of the data. The subset is determined by specifying the percentage of data to sample in every run.

This new feature enables you to use automated hyperparameter tuning to produce optimal settings for selected forecasting parameters to improve forecast accuracy.

A new Hyperparameter Tuning Values column has been introduced on the Forecasting Parameters tab on the Manage Forecasting Profiles page.

New Hyperparameter Tuning Values Column

• The Hyperparameter Tuning Values column provides a list of the values to be tested for the specific forecasting parameter. The hyperparameter tuning logic will test each possible combination of values for the forecasting parameters being tuned and select the ones that provide the best forecast accuracy for each forecasting node (such as item and organization combination.)

• Hyperparameter Tuning Values is a text field where you enter the list of values separated by commas.

• Even if you aren’t sure which list of values to add to which parameter, you can still use the hyperparameter tuning capabilities because this tuning process can be done automatically. The new HypertuneParamSet forecasting parameter contains a default set of parameter values that are used when no values exist for the Hyperparameter Tuning Values field for any of the forecasting parameters.

Additionally, 8 new forecasting parameters have been introduced to configure hyperparameter tuning. These forecasting parameters all appear in the Hyperparameter Tuning category.

New Hyperparameter Tuning Category and Parameters

To enable hyperparameter tuning, you must add the HypertuneSamplePercent parameter to your forecasting profile and change it to a value between 1 and 100.

Details regarding the parameters for hyperparameter tuning

• HypertuneSamplePercent forecasting parameter:
  • Specifies the percentage of the forecast population for which hyperparameter tuning is done during each forecast run. Valid values are integers from 0 to 100.
    • A value of 0 indicates that hyperparameter tuning is disabled.
    • A value of 100 indicates that hyperparameter tuning will be performed for the entire forecast population each time the forecast process is run.
    • Values greater than 0 and less than 100 represent the percentage of the forecast population for which hyperparameter tuning will be performed. In this case, the engine will track forecast nodes that have been tuned and select a different set of nodes each time. For example, if HypertuneSamplePercent=20, then each time the engine is run it will select a different 20% of the forecast population. After 5 runs, 100% of the forecast population will have been covered.
    • The default value is 0.  

NOTE: To enable hyperparameter tuning, you must add the HypertuneSamplePercent forecasting parameter to your forecasting profile and set it to a value greater than 0. This is the only hyperparameter tuning parameters that you must add to your forecasting profile. You can add the other 7 if you want to change the default values.

• HypertuneOutputMode forecasting parameter:
  • Specifies whether to save the results of hyperparameter tuning in the output measures. Valid values are 0, 1, and 2.
    • 0 for no output. Saving only the base, untuned forecast in the forecasting profile's output measure.
    • 1 for tune and use. If the forecast resulting from hyperparameter tuning meets the threshold for forecast accuracy improvement then it will be written to the forecasting profile's output measure along with the tuning settings in the settings measure for hyperparameters otherwise the base, untuned forecast will be output.
    • 2 for writing the base, untuned forecast to the forecasting profile's output measure and the forecast resulting from hyperparameter tuning (if it meets the threshold for forecast accuracy improvement) to the tuned forecast measure for hyperparameters along with the tuning settings in the settings measure for hyperparameters.
    • The default is 2.

• HypertuneHalflife forecasting parameter:
  • Specifies the number of days that the output of hyperparameter tuning is considered fresh. After this period, retuning is done for combinations for which the output of hyperparameter tuning is older than this parameter.
    • Default value is 182 days.

• HypertuneHoldback forecasting parameter:
  • Specifies the number of days before the plan start date to use for holdback validation, that is, out-of-sample testing, of the results of hyperparameter tuning.
    • The value for HypertuneHoldback isn't used if it's greater than the value for HypertuneHoldbackPercent.
    • Default value is 182 days.

• HypertuneHoldbackPercent forecasting parameter:
  • Specifies the maximum percentage of history to use for holdback validation, that is, out-of-sample testing, for each combination during hyperparameter tuning.
    • The value for HypertuneHoldback isn't used if it's greater than the value for HypertuneHoldbackPercent.
    • Default value is 30 percent.

• HypertuneMAPEThreshold forecasting parameter:
  • Specifies the minimum percentage of change required between the base, untuned mean absolute percentage error (MAPE) and the MAPE after hyperparameter tuning for a combination for the optimization to be successful. If the percentage of change is below the threshold, the optimization is ignored.
    • Default value is 1 percent.

• HypertuneParamSet forecasting parameter:
  • Specifies a default set of parameter values to be used for hyperparameter tuning if its enabled, but no values have been entered in the Hyperparameter Tuning Values column on the Forecasting Parameters tab.
  • Default value is: IntermitCriterion[10,90],ForecastValidationSensitivity[0.2,5,10],OutlierSensitivity[2,3,4],CollinearityUseRidge[0,1]

NOTE: This default set of parameter values shouldn’t be edited.

• HypertuneAdvisor forecasting parameter:
  • Currently not used. For future use. Leave the default value of 0.

Hyperparameter tuning output measures

When enabling hyperparameter tuning for a specific forecasting profile, the demand forecast writes the tuning outcome to new output measures.

• If a predefined output measure is selected in the forecasting profile (Bookings Forecast or Shipments Forecast), then the 5 new predefined output measures will be added to the selected measure catalogs in the forecasting profile.

• If a user-defined output measure is selected in the forecasting profile, then the 5 new output measures will be created after you save the profile.

NOTE: For a plan that includes a forecasting profile with a user-defined output measure, you must run the plan with the Refresh with current data option enabled before hyperparameter tuning can be performed on the plan.

Output Measure

New hyperparameter tuning output measures

The names of the hyperparameter tuning output measures vary based on the name of the output measure selected in the forecasting profile. For example, if the output measure is Bookings Forecast, then the hyperparameter tuning output measures will be Bookings Forecast: Tuned, Bookings Forecast: Tuned Settings, Bookings Forecast: Tuned Base MAPE, Bookings Forecast: Tuned Best MAPE, and Bookings Forecast: Last Attempt at Tuning.

• <Forecasting Profile Output Measure name>: Tuned
  • A numeric measure which captures the hyperparameter tuning forecast results.

• <Forecasting Profile Output Measure name>: Tuned Settings
  • A text measure which includes the forecasting parameter Name and the optimized value for each selected parameter for tuning. The measure output will be added for each combination/node based on the related forecast tree (forecasting table).

• <Forecasting Profile Output Measure name>: Tuned Base MAPE
  • The base, untuned out-of-sample MAPE calculated on the plan’s population based on the percent specified in the HypertuneSamplePercent parameter.

NOTE: This is the MAPE calculation on the population before the tuning calculation.

• <Forecasting Profile Output Measure name>: Tuned Best MAPE
  • The best out-of-sample MAPE on the same part of the plan’s population based on the percent specified in the HypertuneSamplePercent parameter.

NOTE: This is the MAPE calculation on the population after the tuning calculation.

• Tuning will occur on nodes where the Tuned Best MAPE < Tuned Base MAPE, which indicates a better forecast accuracy, and when the difference between the 2 MAPES is above the minimum threshold required for tuning. Threshold settings is done on HypertuneMAPEThreshold forecasting parameter. If the percentage of the change is below the threshold, then the optimization is ignored.

• <Forecasting Profile Output Measure name>: Last Attempt at Tuning
  • A date measure to capture the tuning timestamp, so the next time hyperparameter tuning runs, it can decide if it should tune this node or not.

NOTE:

• Hyperparameter tuning populates the attempt date measure for every node it runs on whether or not it actually tuned the node.  There are cases that tuning ran but wasn’t able to improve the forecast. In those cases, the forecast profile populates only the Last Attempt at Tuning with the plan’s run date, without populating the other tuned measures.

• In cases where the Sample Percent is < 100, the combinations that don’t participate in the hyperparameter tuning process will get a Last Attempt at Tuning value of: 01/01/2000.

Example of hyperparameter Tuning Output Measures in a Table After Demand Forecast Run

MAPE process: (using the Bookings Forecast output measure as an example)

• When the demand forecast runs in the Hyperparameter Tuning mode, it includes 3 stages with regards to MAPE:
  • Stage 1: The forecast runs on the full population, and this is the base forecast. MAPE output will be to the Bookings Forecast MAPE measure as always. This MAPE is an in-sample MAPE.
  • Stage 2: The forecast runs on part of the population based on the percent specified in the HypertuneSamplePercent parameter. This is an out-of-sample run and it is before the hyperparameter tuning calculations. Output is to the Bookings Forecast Tuned Base MAPE measure.
  • Stage 3: The forecast calculates the tuning based on the hyperparameter tuning parameters on the same population as in Stage 2 and writes the resulting MAPE to the Bookings Forecast Tuned Best MAPE measure. This is also an out-of-sample MAPE.

• After the 3 stages, the demand forecast compares between the 2 out-of-sample MAPE measures based on the HypertuneMAPEThreshold parameter. This means that if the parameter has 1% as the MAPE threshold, and the Best MAPE is different than the Base MAPE in less than 1%, then the demand forecast won’t write to the to the tuned MAPE measures, because it wasn’t good enough and didn't produce a more accurate forecast.

Example for several forecast demand runs on the same forecasting profile with hyperparameter tuning enabled

Forecasting Profile: HyperTuneSamplePercent=5

• 1st run:
  • Hyperparameter tuning is calculated on a random 5% of the plan’s population, and the tuned output measures will include the output data only on that 5%.
  • On the other 95% of the population, the Bookings Forecast Tuned measure and other tuned measures are all Null, except for the Last Attempt at Tuning which displays the 01/01/2000 date.

• 2nd run:
  • Hyperparameter tuning is calculated on another 5% of the population.

• 3rd run and continue:
  • Same as second run until the HypertuneHalflife number of days is reached. When that stage is reached, then the tuning will retune the existing values from the 1st

Nodal tuning:

The Use Automated Hyperparameter Tuning to Improve Forecast Accuracy feature includes the ability for the demand forecast to use the optimal value per parameter from the Tuned Settings output measure, and to use that measure as the input for the next demand forecast run per node.

2 new forecasting parameters are now available to be used for this purpose.

Nodal Tuning Parameters

• EnableNodalTuning parameter:
  • Specifies whether to use a measure with optimized parameter values as the input for the next time the forecast is generated. If hyperparameter tuning is turned on and HypertuneOutputMode=1, then EnableNodalTuning should be set to 2. Parameter values may be 0, 1, 2.
    • 0 for not using nodal tuning.
    • 1 for always using nodal tuning.
    • 2 for using nodal tuning when hyperparameter tuning is enabled and HypertuneOutputMode=1
    • Default value is 2.

• SetNodalTuningSource Parameter:
  • Currently not used. For future use.

Steps to Enable

You don't need to do anything to enable this feature.

Tips And Considerations

• Hyperparameter related forecasting parameters will be used only in cases where the HyperTuneSamplePercent is > 0.

• The HypertuneHoldbackPercent parameter is an optional parameter, and it’s a secondary limit to the HypertuneHoldback parameter as the max % of history to use as holdback for each node. When using this parameter as well, the holdback used for each combination eventually will be smaller from the values of the HypertuneHoldback and the HypertuneHoldbackPercent.

• 10 predefined measures were added in in this update:
  • Bookings Forecast: Tuned Settings
  • Shipments Forecast: Tuned Settings
  • Bookings Forecast: Tuned
  • Shipments Forecast: Tuned
  • Bookings Forecast: Tuned Base MAPE
  • Shipments Forecast: Tuned Base MAPE
  • Bookings Forecast: Tuned Best MAPE
  • Shipments Forecast: Tuned Best MAPE
  • Bookings Forecast: Last Attempt at Tuning
  • Shipments Forecast: Last Attempt at Tuning

Key Resources

• Watch the Use Automated Hyperparameter Tuning to Improve Forecast Accuracy demo.

Access Requirements

There are no new roles or security privileges needed to access this feature.

• Users who currently have access to the Manage Forecasting Profiles task will be able to use this feature.

Users who are assigned a configured job role that contains these privileges can access this feature:

• Edit Forecasting Profiles (MSC_EDIT_FORECASTING_PROFILES_PRIV)
• Edit Plans (MSC_EDIT_PLANS_PRIV)
• Edit Plan Options (MSC_EDIT_PLAN_OPTIONS_PRIV)

These privileges were available prior to this update.