Skip to main content
Cross Validated

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

4
  • $\begingroup$ The cut between training and test data is (in most cases) as arbitrary as any cut inside the training set. Additionally, what excactly does it tell you if your accuracy/any other measure is slightly worse on the test set? And if it is significantly worse, all you know is something went wrong with your CV. Anyways, for both arguments, doing a repeated CV would give you more robust results (with each individual CV being better). If computational expense is a factor, comparing variance should give a sufficiently good measure in any of those cases without the risk of an (un)lucky test set. $\endgroup$ Commented Nov 11, 2024 at 13:57
  • $\begingroup$ @Linrael You say "if it is significantly worse, all you know is something went wrong with your CV", but without a test set you would not have realised this. How you separate the test data and create the cross-validation folds also matters: if you are looking at multiple observation for various individuals, there is a distinction between forecasting for the same individuals or forecasting fro new individuals, ans your data splits should reflect this. $\endgroup$ Commented Nov 11, 2024 at 21:53
  • $\begingroup$ Usually a repeated CV shows this. And to include measurement specific features into your training process there is grouped CV to consider this. $\endgroup$ Commented Nov 15, 2024 at 12:38
  • $\begingroup$ It is a best practice. I have definitely at times ignored this and just relied on CV results, because, indeed, I also believe it is usually fine. There are times when your CV results will however overestimate true performance, when you have little data, lots of hyperparameters. And it is not so easy to know when this starts having an effect. $\endgroup$ Commented Nov 18, 2024 at 8:03