Convergence Check
Convergence checks ensure optimization processes reach optimal solutions, stopping when performance criteria are met.
In L-BFGS solver, the convergence criteria includes maximum number of
iterations, infinity norm of gradient, and relative error tolerance. For held-aside
regularization, the convergence criteria checks the loss function value of the test data
set, as well as the best model learned so far. The training is terminated when the model
becomes worse for a specific number of iterations (specified by
NNET_HELDASIDE_MAX_FAIL
), or the loss function is close to zero, or
the relative error on test data is less than the tolerance.