Skip to main content
Cross Validated

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

12
  • 18
    $\begingroup$ +11! Finally an answer with an explicit simulation! And it goes directly against the conclusion of the currently accepted and of the most upvoted answers. Regarding your conclusion: if indeed "the model stability is a key factor", then one should be able to set up a simulation where the variance would increase with $K$. I've seen two simulations: yours here, and this one and both show that the variance either decreases or stays constant with $K$. Until I see a simulation with increasing variance, I'll remain very skeptical that it ever does. $\endgroup$ Commented Jul 18, 2018 at 12:06
  • 7
    $\begingroup$ @amoeba here's a case where LOOCV fails: consider n data points and an interpolating polynomial of degree n. Now double the number of data points by adding a duplicate right on each existing point. LOOCV says the error is zero. You need to lower the folds to get any useful info. $\endgroup$ Commented Jul 18, 2018 at 14:15
  • 2
    $\begingroup$ For thos interested in this discussion - lets continue in chat: chat.stackexchange.com/rooms/80281/… $\endgroup$ Commented Jul 20, 2018 at 7:46
  • 3
    $\begingroup$ Have you considered the fact that $k$-fold with e.g. $k=10$ allows repetition? This is not an option with LOOCV, and thus should be taken into account. (Repetition of the k-fold partitioning and procedure with the same sample.) $\endgroup$ Commented Jul 20, 2018 at 9:40
  • 2
    $\begingroup$ @amoeba: re Kohavi/ LOO and variance. I found that LOO for some classification models can be quite (surprisingly) unstable. This is particularly pronounced in small sample size, and I think it is related to the test case always belonging to the class that is underrepresented wrt. the whole sample: in binary classification stratified leave-2-out does not seem to have this problem (but I did not test extensively). This instability would add to the observed variance, making LOO stick out of the other choices of k. IIRC, this is consistent with Kohavi's findings. $\endgroup$ Commented Jul 23, 2018 at 17:27