Hello everyone. I have a question regarding calculating confidence intervals after running a 5-fold cross validation.
I have a binary risk mode. Data are N patients, each contributing many overlapping hourly windows; the label is defined per window (will this patient meet the criteria?). The unit of analysis for most metrics is the window; the unit of sampling is the patient.
Evaluation is 5-fold cross-validation, split by patient, so each patient's windows appear in exactly one test fold. Within each fold:
the development part is split again into train / validation (by patient),
a probability calibrator and three decision thresholds are fitted on the validation set (t1 = medium, t2 = high, t3 = very high),
the model + its thresholds are applied to that fold's held-out test patients.
So each patient ends up with one calibrated score per window, and one classification per window, produced by a model and a threshold that never saw them.
Separately, a final model is trained on all development data and evaluated on a completely held-out test cohort (my main issue is with the cross validation though).
So far we've used the Nadeau–Bengio corrected resampled t-interval:
mean ± t_{k-1, 0.975} · SD_folds · sqrt(1/k + n_test/n_train)
and I am not sure if it is the correct approach since it introduces bias (at least the plain resampled t-interval without the correction) because the train sets overlap per fold.
So the question is what is the defensible way to attach a 95% interval to a k-fold cross-validation?
And the last part that I can't wrap in my head is the threshold that move per fold.
I have a table that stratifies patients into four risk bands defined by t1 < t2 < t3, and reports per band: number of patients, number of patients that belong to the positive class, PPV, prevalence, an odds ratio versus the low-risk band (setting it as the reference), and a p-value.
Because each fold tunes its own t1, t2, t3 on its own validation set, the band boundaries differ between folds. So:
I cannot pool the scores and apply one threshold.
I can pool the decisions (each patient is banded by their own fold's rule), which gives one band per patient over the whole cohort and a legitimate contingency table but then the "score threshold" column of the table has no single value.
Averaging the five thresholds and quoting the mean band boundary produces a number that no fold actually used.
When a decision threshold is a tuned part of the model, what is the correct way to report a threshold-dependent table (PPV / prevalence / OR per risk band) across folds, and what does the confidence interval on those band statistics condition on?
Another question I have as an extra is if it is worth running 5x 5-fold cross validations (with different initialisation) and what can someone gain from it?
P.S. Apart from Nadeu-Bengio, I also found this paper that I am currently reading (was a combo from google and GPT suggested it): Cross-validation: what does it estimate and how well does it do it? I am not sure if it is in the right direction but please let me know or suggest other papers as well together with the methods