Why does LOOCV yield test estimates with higher variance, while it should remain the same each time?

by DataFace   Last Updated August 28, 2019 18:19 PM

Why do the authors of Introduction to Statistical Learning state that:

..the test error estimate resulting from LOOCV tends to have higher variance than does the test error estimate resulting from k-fold CV.

while LOOCV clearly yields the same results each time, thus resulting in no variance at all?

I know formulations of this question have already been answered and I read them all, but I just don't get what we're looking at. I've been trying to come up with some adequate code that can show why this is the case... Here's what I did:

library(ISLR)
library(boot)

CV <- function(folds) {
  it <- 10 #number of iterations

  cv.err <- rep(0.0, it)
  for(i in 1:it) {
    lm.fit <- glm(mpg~poly(horsepower,2), data = Auto)
    cv.err[i] <- cv.glm(Auto, lm.fit, K = folds)$delta[1]
  }
  cv.err
}

folds <- c("K2" = 2, "K5" = 5,"K10" = 10, "K20" = 20,
           "K50"=50, "K100" = 100, "LOOCV" = nrow(Auto))

cv.folds <- matrix(rep(NA,7*10), 
                   nrow = 7, 
                   dimnames = list(names(folds), 1:10))

for(k in 1:length(folds)) {
  cv.folds[k,] <- CV(folds[k])
}
cv.folds
apply(cv.folds, 1, var)

As I expected increasing K, only makes the test error estimate less variant. Can someone please help me understand what is exactly meant by higher variance (and what exactly is it that has it), because clearly I'm interpreting it wrong.

Thanks in advance!



Related Questions



Cross validation and the Bias Variance trade-off

Updated November 15, 2018 09:19 AM

Machine Learning Model Evaluation

Updated May 22, 2019 22:19 PM


Bias/variance tradeoff tutorial

Updated October 04, 2018 19:19 PM