How Hyperparameter Tuning Is Done

This topic explains how hyperparameter tuning is done for the forecasting parameters of a user-defined forecasting profile of the Bayesian machine-learning type for a demand or demand and supply plan.

If the HypertuneSamplePercent forecasting parameter has been added to the forecasting profile and set to a value greater than 0 (zero), hyperparameter tuning will happen as follows:

The forecast is calculated according to the forecasting parameters of the forecasting profile for the entire plan population.
The mean absolute percentage error (MAPE) is calculated and stored in the <predefined output measure> MAPE or <user-defined output measure> MAPE measure as usual. This calculation is for the in-sample MAPE.
For the demand history before the plan start date that's determined by the values of the HypertuneHoldback and HypertuneHoldbackPercent forecasting parameters, for the percentage of forecasting nodes (combinations) specified by the HypertuneSamplePercent forecasting parameter, the forecast is calculated according to the forecasting parameters of the forecasting profile.
The calculated MAPE is stored in the Bookings Forecast: Tuned Base MAPE, Shipments Forecast: Tuned Base MAPE, or <user-defined output measure>: Tuned Base MAPE measure for the forecasting profile. This calculation is for the out-of-sample base MAPE.
For the forecasting nodes in Step 2, hyperparameter tuning is performed, and the MAPE for the forecast according to the tuned forecasting parameters is stored in the Bookings Forecast: Tuned Best MAPE, Shipments Forecast: Tuned Best MAPE, or <user-defined output measure>: Tuned Best MAPE measure for the forecasting profile. This calculation is for the out-of-sample best MAPE.
If the HypertuneMaxPermutations forecasting parameter is set to a value greater than 1, a random-walk algorithm is used for selection of the best values for forecasting parameters from the values provided in the Hyperparameter Tuning Values column in the Parameters step in the guided process for creating or editing a forecasting profile. If no values are available in the column, the default values in the HypertuneParamSet forecasting parameter are used for selection of the best values. For more information, see the topics titled How You Improve the Performance of Hyperparameter Tuning and Forecasting Parameters for Hyperparameter Tuning.
If the HypertuneMaxPermutations forecasting parameter is set to 0 (zero) or 1, all permutations are evaluated for the values provided in the Hyperparameter Tuning Values column in the Parameters step in the guided process for creating or editing a forecasting profile. If no values have been provided, the default set of values in the HypertuneParamSet forecasting parameter is used for evaluation of the permutations.
Holdback validation (out-of-sample testing) is performed.
Hyperparameter tuning is considered as successful if both of these conditions are met:
- The tuned best MAPE is less than the tuned base MAPE.
  This situation indicates improved forecast accuracy.
- The difference between the tuned best MAPE and tuned base MAPE is equal to or more than the value specified in the HypertuneMAPEThreshold forecasting parameter.
If the value of the HypertuneOutputMode forecasting parameter is 1, for the successfully tuned forecasting nodes, the forecast is calculated according to the tuned forecasting parameters and stored in the output measure for the forecasting profile. The Bookings Forecast: Tuned, Shipments Forecast: Tuned, or <user-defined output measure>: Tuned measure isn't updated. For the remaining forecasting nodes, the forecast according to the forecasting profile's forecasting parameters is written to the output measure.
If the value of the HypertuneOutputMode forecasting parameter is 2, for the successfully tuned forecasting nodes, the forecast is calculated according to the tuned forecasting parameters and stored in the Bookings Forecast: Tuned, Shipments Forecast: Tuned, or <user-defined output measure>: Tuned measure. For all the forecasting nodes, the forecast according to the forecasting profile's forecasting parameters is stored in the output measure.
The data between the plan start date and plan end date is considered for the forecast that's calculated according to the tuned forecasting parameters for the successfully tuned forecasting nodes.
For all the forecasting nodes that participated in hyperparameter tuning (whether successful or not), the results of hyperparameter tuning are stored in the tuned-settings measure. The names and optimized values for the tuned forecasting parameters are also stored in the measure along with the levels of the forecasting nodes in the forecast tree. If the values in the HypertuneParamSet forecasting parameter are used, then the selected values from this forecasting parameter are stored in the tuned-settings measure.
For more information, see the topic titled Text in the Tuned-Settings Measure for Hyperparameter Tuning.
For all the forecasting nodes that participated in hyperparameter tuning (whether successful or not), the date and time of hyperparameter tuning are stored in the Bookings Forecast: Last Attempt at Tuning, Shipments Forecast: Last Attempt at Tuning, or <user-defined output measure>: Last Attempt at Tuning measure.
For the remaining forecasting nodes, the value of the measure is 01/01/2000.
If the value of the HypertuneAdvisor forecasting parameter is 1, and forecasting nodes have been successfully tuned, a notification is provided through the Planning Advisor.
For more information, see the topic titled How You Set Up Hyperparameter-Tuning Notifications in the Planning Advisor.
During the next plan run, Steps 1 through 8 are repeated.
A different set of forecasting nodes for hyperparameter tuning is selected according to the value of the HypertuneSamplePercent forecasting parameter. The date in the Bookings Forecast: Last Attempt at Tuning, Shipments Forecast: Last Attempt at Tuning, or <user-defined output measure>: Last Attempt at Tuning measure and value of the HypertuneHalflife forecasting parameter are considered for determining if a forecasting node is eligible for hyperparameter tuning.
If the value of the EnableNodalTuning forecasting parameter is 0 (zero), the values in the base MAPE, best MAPE, and tuned forecast measures for the forecasting nodes processed during the previous plan run are retained until the forecasting nodes are tuned again. For these forecasting nodes, the forecasting parameters in the forecasting profile will apply.
If the value of the EnableNodalTuning forecasting parameter is 1 or 2, nodal tuning is performed, and the forecasting parameters tuned during the previous plan run are applied to the forecasting nodes in later plan runs until the value of the HypertuneHalflife forecasting parameter is reached. For more information, see the topic titled How You Enable Nodal Tuning for Forecasting Nodes.

Other Points About the Process for Hyperparameter Tuning

Note these points about hyperparameter tuning:

The forecasting nodes for hyperparameter tuning are selected from the lowest level of the forecast tree, and this level can be aggregated.
If hyperparameter tuning can't happen for a forecasting node at the lowest level of the forecast tree, but the forecast according to the forecasting parameters of the forecasting profile is successful for the forecasting node, then hyperparameter tuning isn't done for the forecasting node.
If the forecast according to the forecasting parameters of the forecasting profile fails for the forecasting node, then hyperparameter tuning is attempted for the parents of the forecasting node while a forecast is generated for the parent. The level at which the hyperparameter tuning is done is reflected in the Bookings Forecast: Tuned Settings, Shipments Forecast: Tuned Settings, or <user-defined output measure>: Tuned Settings measure.
If the base MAPE can't be calculated for a forecasting node but the best MAPE is calculated, the tuned forecasting parameters are accepted, and the tuned forecast is written to the output or tuned measure as specified by the HypertuneOutputMode forecasting parameter. The tuned base MAPE measure won't have a value, but the tuned best MAPE measure will have a value.
In the tuned-settings measure for the forecasting node, the following text is populated: "Tuned - Improvement found, base failed"
These forecasting nodes aren't included in the average calculations returned through the Planning Advisor for hyperparameter tuning.
When nodal tuning isn't enabled, and the value of the HypertuneSamplePercent forecasting parameter is 100, then the results of hyperparameter tuning from previous plan runs are cleared from the measures.
When nodal tuning isn't enabled, and hyperparameter tuning is performed again for a previously tuned forecasting node, the tuned settings in the tuned-settings measure are overwritten with the newly tuned settings.
If nodal tuning is enabled, you change the hyperparameter-tuning values for forecasting parameters after a tuning run, and you perform hyperparameter tuning again, the values for previously tuned forecasting parameters that aren't in the current set of tuned forecasting parameters are retained in the tuned-settings measure and used. For tuned forecasting parameters that are changed, the old values are overwritten with new values in the tuned-settings measure.