In holdout forecasting:
The last few data points are removed from the data series. The remaining historical data series is called in-sample data, and the holdout data is called out-of-sample data. Suppose p periods have been removed as holdout from a total of N periods.
Parameters are optimized by minimizing the fit error measure for in-sample data. If method parameters are provided by the user, those are used in the final forecasting.
After the parameters are optimized, the forecasts for the holdout periods (p periods) are calculated.
The error statistics (RMSE, MAD, MAPE) are out-of-sample statistics, based on only the numbers in the hold-out period. The RMSE for holdout forecasting is often called holdout RMSE. The holdout error measures are the ones reported to the user and are used to sort the forecasting methods.
Other statistics such as Theil's U, Durbin-Watson, and Ljung-Box are in-sample statistics, based on the non-holdout period.
Final forecasting is performed on both the in-sample and out-of-sample periods (all N periods) using the standard technique.
The standard error for the forecasts is also calculated using all N periods.
To improve the optimized parameter values obtained for the method, holdout forecasting should be used only when there are at least 100 data points for non-seasonal methods and 5 seasons for seasonal methods. For best results, use no more than 5 percent of the data points as holdout, no matter how large the number of total data points.