Text in the Tuned-Settings Measure for Hyperparameter Tuning
This topic explains the text in the tuned-settings measure for hyperparameter tuning of your demand or demand and supply plan.
When you run your plan that's configured for hyperparameter tuning, information about the results of hyperparameter tuning for each forecasting node (combination) is stored in the Bookings Forecast: Tuned Settings, Shipments Forecast: Tuned Settings, or <user-defined output measure>: Tuned Settings measure. You can use this measure to analyze the results of hyperparameter tuning for the forecasting nodes in your plan.
Results of Hyperparameter Tuning
This table lists and explains the text that appears in the tuned-settings measure after the first plan run with hyperparameter tuning:
| Text | Result | Meaning |
|---|---|---|
| Tuned - Improvement found above threshold | Success | The forecasting parameters were successfully tuned for the
forecasting node. The improvement to the forecast accuracy equaled or exceeded the value in the HypertuneMAPEThreshold forecasting parameter. The tuned-settings measure will contain the names and optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Improvement found, base failed | Success | The base mean absolute percentage error (MAPE) couldn't be found
for the forecasting node, but the forecasting parameters were
successfully tuned. The tuned-settings measure will contain the names and optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Base is optimal | Success | The forecasting-parameter values in the forecasting profile are
optimal for the forecasting node. The tuned-settings measure will contain the forecast-tree level for the forecasting node. |
| Failed - on all tested parameter values | Failure | Hyperparameter tuning couldn't be done for the forecasting node
for all permutations of the hyperparameter-tuning values. The tuned-settings measure will contain the forecast-tree level for the forecasting node. |
| Tuned - No improvement, all params have no impact | Failure | No changes were identified to the forecasting-parameter values
for the forecasting node that affected the forecast accuracy. The tuned-settings measure will contain the forecast-tree level for the forecasting node. |
| Failed - not enough history for tuning | Failure | Hyperparameter tuning couldn't be done for the forecasting node
because there wasn't enough history for holdback validation
(out-of-sample testing). The tuned-settings measure will contain the forecast-tree level for the forecasting node. |
| Tuned - No improvement above threshold, <number>% found | Failure | Changes were identified for the forecasting-parameter values for
the forecasting node. However, the improvement to the forecast accuracy didn't equal or exceed the value in the HypertuneMAPEThreshold forecasting parameter. The tuned-settings measure will contain the forecast-tree level for the forecasting node. |
| Tuned - No improvement above threshold, Base is best <number>% | Failure | The forecasting-parameter values in the forecasting profile are
optimal for the forecasting node. The base MAPE for holdback validation (out-of-sample testing) was the most accurate MAPE, but the improvement to the forecast accuracy didn't equal or exceed the value in the HypertuneMAPEThreshold forecasting parameter. The tuned-settings measure will contain the forecast-tree level for the forecasting node. |
| Tuned - Improvement above threshold but full forecast failed. Reset to Base | Failure | The forecasting parameters were successfully tuned for the
forecasting node, and the improvement to the forecast accuracy
equaled or exceeded the value in the HypertuneMAPEThreshold
forecasting parameter. However, the tuned forecast failed. Therefore, the base forecast according to the forecasting parameters of the forecasting profile is being used for the forecasting node. The tuned-settings measure will contain the forecast-tree level for the forecasting node. |
This table lists and explains the text that appears in the tuned-settings measure after subsequent plan runs with hyperparameter tuning:
| Text | Result | Meaning |
|---|---|---|
| Tuned - Prior is best | Success | The tuned settings from the previous tuning run for the plan are
optimal for the forecasting node. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Improvement found above threshold | Success | The forecasting parameters were successfully tuned for the
forecasting node. The improvement to the forecast accuracy equaled or exceeded the value in the HypertuneMAPEThreshold forecasting parameter. The tuned-settings measure will contain the names and optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Improvement found, base failed | Success | The base MAPE couldn't be found for the forecasting node, but the
forecasting parameters were successfully tuned. The tuned-settings measure will contain the names and optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Base is best,prior parameters were reset: c=#, d=#, ... | Success | Hyperparameter tuning wasn't successful in the present plan run
but was successful in the previous plan run. The forecasting-parameter values of the forecasting profile are optimal for the forecasting node. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Base is best but <number>% below threshold compared to prior,keep prior | Success | The forecasting-parameter values of the forecasting profile are
optimal for the forecasting node. However, the difference in the forecast accuracy of the forecasting profile and the prior settings didn't equal or exceed the value of the HypertuneMAPEThreshold forecasting parameter. Therefore, the prior settings are being retained. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - No improvement, all params have no impact, keeping prior | Failure | No best permutation could be found from the hyperparameter-tuning
values. Therefore, the tuned settings from the previous plan run are
being retained. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - No improvement above threshold, <number>% found, keeping prior | Failure | The forecasting parameters were tuned for the forecasting
node. However, the improvement to the forecast accuracy didn't equal or exceed the value in the HypertuneMAPEThreshold forecasting parameter. Therefore, the tuned settings from the previous plan run are being retained. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Improvement above threshold but full forecast failed, keeping prior | Failure | The forecasting parameters were successfully tuned, and the
improvement to the forecast accuracy equaled or exceeded the value
in the HypertuneMAPEThreshold forecasting parameter. However, the tuned forecast failed. Therefore, the tuned settings from the previous plan run are being used. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Failed on all parameter values, keeping prior | Failure | All permutations of the hyperparameter-tuning values failed for
the forecasting node. The tuned settings from the previous plan run
are being retained. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - Base is best but <number>% below threshold, keeping prior | Failure | The forecasting-parameter values of the forecasting profile are
optimal for the forecasting node. However, the difference in the forecast accuracy of the forecasting profile and the prior settings didn't equal or exceed the value of the HypertuneMAPEThreshold forecasting parameter. Therefore, the prior settings are being retained. This text appears only when nodal tuning is enabled, and incremental tuning is being done. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
| Tuned - not enough history for tuning, keeping prior | Failure | Hyperparameter tuning couldn't be done for the forecasting node
because there wasn't enough history for holdback validation
(out-of-sample testing). The tuned settings from the previous plan
run are being retained. This text appears only when nodal tuning is enabled. The tuned-settings measure will contain the names and previously optimized values for the tuned forecasting parameters and the forecast-tree level for the forecasting node. |
This table lists and explains other text that appears in the tuned-settings measure:
| Text | Result | Meaning |
|---|---|---|
| Wrong parameter tuning instruction: ParamTuner: Disabled due to wrong settings | Failure | Hyperparameter tuning couldn't be done for forecasting nodes because values were incorrectly provided in the Hyperparameter Tuning Values column in the Parameters step in the guided process for creating or editing the forecasting profile. |
| Not Tuned - Level change,Fl=n,prior setting | Failure |
During the previous tuning run, hyperparameter tuning was successful at a particular level of the forecast tree. However, in the present tuning run, the forecast succeeded at another level of the forecast tree, and hyperparameter tuning wasn't successful. The tuned-settings measure will contain the new forecast-tree level for the forecasting node. |
Level for Hyperparameter Tuning
The tuned-settings measure indicates the level in the forecast tree for the forecasting node for which hyperparameter tuning was done. This information is provided in the FL=<number> format, where FL is the initialism for forecasting level, and number is the level in the forecast tree. When the number is 1, hyperparameter tuning has been done at the lowest level of the forecast tree.