I am trying to understand how bias and variance of a resampling scheme like cross validation are different from bias and variance of a model (like a linear model or a tree).
Which of the following is true (if any)?
Bias and variance of cv are the bias and variance of the accuracy estimate obtained from using cv as a resampling scheme. Because of this, bias and variance of cv tells us something about the accuracy of the estimate (variance: like the standard error of an estimator in statistics, its precision) and whether the estimate will converge towards the real and unknown out-of-sample error, for example, if the sample is large enough (then bias=0).
Because of 1.: CV itself can have small bias and small variance, but the predicted error can have large bias and variance (it is just that the CV estimates it well. But the estimated error itself is large. That is something else).
If 1. and 2. are true, then: why is large variance of a resampling scheme a problem? I can only imagine that resulting confidence intervals are large, which might make it hard to find differences between models.
I understand bias variance decomposition and dilemma in the context of model selection, and that we will never see variance and bias in real data. We have to estimate the error, for example using cv.
Kind regards Joern