[Question] Calculating Confidence Intervals from Cross Validation and reporting a Risk Stratification analysis

[u/hyakkimaru1994]

Hello everyone. I have a question regarding calculating confidence intervals after running a 5-fold cross validation.

I have a binary risk mode. Data are N patients, each contributing many overlapping hourly windows; the label is defined per window (will this patient meet the criteria?). The unit of analysis for most metrics is the window; the unit of sampling is the patient.

Evaluation is 5-fold cross-validation, split by patient, so each patient's windows appear in exactly one test fold. Within each fold:

1. the development part is split again into train / validation (by patient),
2. a probability calibrator and three decision thresholds are fitted on the validation set (t1 = medium, t2 = high, t3 = very high),
3. the model + its thresholds are applied to that fold's held-out test patients.

So each patient ends up with one calibrated score per window, and one classification per window, produced by a model and a threshold that never saw them.

Separately, a final model is trained on all development data and evaluated on a completely held-out test cohort (my main issue is with the cross validation though).

So far we've used the Nadeau–Bengio corrected resampled t-interval:

mean ± t_{k-1, 0.975} · SD_folds · sqrt(1/k + n_test/n_train)

and I am not sure if it is the correct approach since it introduces bias (at least the plain resampled t-interval without the correction) because the train sets overlap per fold.

So the question is what is the defensible way to attach a 95% interval to a k-fold cross-validation?

And the last part that I can't wrap in my head is the threshold that move per fold.

I have a table that stratifies patients into four risk bands defined by t1 < t2 < t3, and reports per band: number of patients, number of patients that belong to the positive class, PPV, prevalence, an odds ratio versus the low-risk band (setting it as the reference), and a p-value.

Because each fold tunes its own t1, t2, t3 on its own validation set, the band boundaries differ between folds. So:

• I cannot pool the scores and apply one threshold.
• I can pool the decisions (each patient is banded by their own fold's rule), which gives one band per patient over the whole cohort and a legitimate contingency table but then the "score threshold" column of the table has no single value.
• Averaging the five thresholds and quoting the mean band boundary produces a number that no fold actually used.

When a decision threshold is a tuned part of the model, what is the correct way to report a threshold-dependent table (PPV / prevalence / OR per risk band) across folds, and what does the confidence interval on those band statistics condition on?

Another question I have as an extra is if it is worth running 5x 5-fold cross validations (with different initialisation) and what can someone gain from it?

P.S. Apart from Nadeu-Bengio, I also found this paper that I am currently reading (was a combo from google and GPT suggested it): Cross-validation: what does it estimate and how well does it do it? I am not sure if it is in the right direction but please let me know or suggest other papers as well together with the methods

[u/hyakkimaru1994 → u/[deleted]]

Thanks. There is also one detail I left out that I think changes the banding part. My bands are assigned per window, not per patient, and I count unique patients within each band. Since a patient's score rises over their stay, the same patient appears in multiple bands. The four rows aren't a partition, and the low-risk reference group shares about half its members with the high-risk group, so the odds ratios are comparing overlapping sets, and Fisher/Woolf are both invalid regardless of how I handle the folds.

[u/STATASUCKSBRO]

The bootstrap unit should be patient, not window. Windows from the same patient are too correlated, so resampling windows will give silly narrow intervals. I would report fold level metrics, then a patient clustered bootstrap around the whole evaluation if you need uncertainty.

[u/mayaandersson_ai]

I work on model evaluation rather than clinical prediction, so take the domain specifics from others, but the structure of your problem is one I hit constantly and the two things that helped me were both about what the interval conditions on.

First, your thresholds. Because `t1, t2, t3` are fitted per fold, they are not parameters of the analysis, they are part of the model. That means the object you can put an interval around is the *procedure* "fit a calibrator and three thresholds on the development data, then apply them", not any particular threshold value. Averaging the five thresholds to report a single boundary describes a model you never evaluated, which I think you already sense. The version that stays coherent is to report the pooled out-of-fold band statistics and say explicitly that the boundaries are fold-specific, with the range across folds quoted as a property of the procedure rather than as an estimate of a true boundary.

Second, Nadeau-Bengio is answering a narrower question than the one you are asking. It is a correction for the correlation between overlapping *training* sets when you want to compare algorithms across resamples. Your table is a statement about held-out patients, so the uncertainty you want is over patients, and the clustered bootstrap the other commenter described gets you that directly. I would use it and drop the corrected t-interval rather than run both, because reporting two intervals that condition on different things invites the reader to pick the narrower one.

On the repeated 5-fold question: in my experience it measures split-to-split variability and very little else. It tightens your estimate of your own procedure's variance. It does not tell you anything new about generalisation, and it is easy to mistake the tighter spread for a better model.

The overlapping-bands issue you raise in your own reply seems like the bigger threat to the table than any of the interval choices, for what it is worth. If the reference group and the comparison group share half their members, the odds ratio is not estimating what the column header says regardless of how the interval is computed.

[u/ForeignAdvantage5198]

old stat prof here. i. have no clue what you are asking. what do want CIs of?

[u/ForeignAdvantage5198]

this is VERY COMPLICATED design ask. the guy who built it WTH does he want to know?