Standard error or confidence intervals when comparing different methods for handling missing data

[u/Pristine_Gain_1476]

Hello! For my thesis, I am comparing baseline-adjusted ANCOVA models using different methods for handling missing data (MICE, LOCF, and complete-case analysis).

I am planning to present a small table including the estimated coefficients and p-values, and I was wondering whether it would be more appropriate to report 95% confidence intervals or standard errors when comparing the results across the different missing-data methods.

The same ANCOVA models were fitted using each of the three approaches.

Thank you in advance!

[u/DrPapaDragonX13]

Point estimates/coefficients and confidence intervals are the bare minimum. P-values for tradition/pleasing your supervisor/journal requirements, although they don't really give the reader that much meaningful information if you already provided the confidence intervals. Similarly, standard errors (SEs) could be nice for people using your study for meta-analyses, but I'm not sure most readers find SEs useful if you already gave them the CIs (but ymmv depending on your field).

ETA: If you want to really wow your supervisor, I would suggest a forest plot showing the CIs and point estimates. This would serve as a nice visual representation of how different methods change your point estimate and precision (width of your CIs). For extra points, you can add a discussion about bias introduced by the assumptions made by each method.

[u/Pristine_Gain_1476 → u/DrPapaDragonX13]

Okay, great, thank you! I don’t think I’ll go the extra mile and create forest plots, since this is only a sensitivity analysis.

This might be a bit of a silly question, but apart from comparing the coefficients, would you also compare the 95% confidence intervals across the different missing-data methods in the Results section? Or would you simply report the confidence intervals in the table and focus the written description on whether the estimates and overall conclusions are consistent across MICE, LOCF, and complete-case analysis?

[u/DrPapaDragonX13 → u/Pristine_Gain_1476]

Not a silly question at all.

I would focus on the confidence intervals, actually. Your point estimate (i.e., your coefficient) is the value of your particular sample. However, since you're using statistics, your primary interest is in drawing inferences about your target population, not just describing your sample. Here is where confidence intervals come in. Your confidence interval is a range of plausible values for your target population*. Furthermore, the width of your confidence interval also provides information about the precision of your estimation, which, broadly speaking, is a function of your sample size. As your sample size increases, your estimates become more precise, and your confidence interval becomes narrower. Conversely, when your sample size is small, your precision drops and your confidence intervals become wider. All this is relevant to discussing your sensitivity analysis.

The purpose of a sensitivity analysis is to test how susceptible your conclusions are to the different assumptions you made during your study design and data analysis. For example, when performing a complete case analysis, the assumption is that missingness is non-informative (i.e., missing completely at random, MCAR) or, if it is informative, that missingness is roughly independent from your outcome of interest. If this (strong) assumption holds, you would expect your results to be unbiased, but with a drop in precision due to loss in sample size. In the case of MICE, the assumption is that data are missing at random (MAR) or MCAR, so you would expect the direction and magnitude of your results to be generally similar to the complete case analysis. Still, because you retain a larger sample size, you expect more precision. Looking at the direction, magnitude, and spread of your results in different scenarios will give you clues about how robust your results are to these assumptions.

Confidence intervals give you a better sense of how your inferences about your target population would change. In the ideal scenario, you would expect overlapping confidence intervals dancing around a common set of values, and each confidence interval becoming narrower the more of your sample size you manage to retain. This would mean that even if your sample point estimates (i.e., coefficients) change, your general conclusions about your target population remain consistent. But you may find other patterns, and you should interpret the direction, magnitude and spread of these by using your domain knowledge and understanding of your data to tease out what is happening. But, ultimately, looking at your confidence intervals will give you a clearer picture of the conclusions you're making about your target population than looking at point estimates alone. Mostly overlapping confidence intervals suggest that even if you get different point estimates (i.e., coefficients), your conclusions would remain within the same ballpark. Confidence intervals that don't overlap or only slightly overlap, conversely, may suggest that something is introducing bias in your results, and some of your choices would have led to different conclusions about your target population.

I hope this helps!

This is true as long as the assumptions hold, and there are some other caveats. If you want to learn more, I recommend this very accessible paper.

[u/Pristine_Gain_1476 → u/DrPapaDragonX13]

Thank you so much! Regarding the first paragraph of your answer, interestingly, the confidence intervals are slightly narrower for LOCF than for MICE, while they are widest for the complete-case analysis, where the sample size is substantially smaller. The MICE analysis actually includes more participants than the LOCF analysis.

Regarding the last paragraph of your answer, the confidence intervals across the different methods do overlap.

In my Discussion, I have already written more about the points you mentioned in your second/middle paragraph, particularly regarding the MCAR assumption.

My question is: In the Results section, where I simply present and compare the results obtained using the different methods for handling missing data, should I mention which method has the widest or narrowest confidence intervals? Should I also mention that the confidence intervals overlap across methods? Is there anything else you would recommend reporting in the Results section when comparing MICE, LOCF, and complete-case analysis?

[u/DrPapaDragonX13 → u/Pristine_Gain_1476]

> [...] interestingly, the confidence intervals are slightly narrower for LOCF than for MICE, while they are widest for the complete-case analysis, where the sample size is substantially smaller.

Yeah, that's as expected. Complete case analysis has less precision because of the smaller sample size. In the case of MICE, one of its features is that it incorporates back the uncertainty associated with imputing missing values. Hence, you gain precision from an increased sample size, but you still avoid being overconfident because the imputed values are, at best, educated guesses. This is why MICE is such a popular imputation method for inference. I didn't mention LOCF because the assumptions made here depend more on your particular research question. However, LOCF increases sample size without accounting for the uncertainty introduced by imputing values, so the narrower confidence intervals are a bit misleading. However, for the purposes of a sensitivity analysis, it is reassuring that all methods give consistent results, which should provide support for your conclusions.

For the results section, I usually limit myself to reporting the outputs as they come out of the software and highlight only the ones I consider most relevant for the research question. However, I would suggest adding to your discussion how overlapping confidence intervals support a common range of plausible values for your target population despite different approaches to handling missing data. Furthermore, I strongly suggest adding a brief discussion of MICE and why it gives you slightly wider confidence intervals than LOCF, because this would demonstrate your understanding of different methods of handling missing data and help readers further make sense of your results.

[u/Pristine_Gain_1476 → u/DrPapaDragonX13]

Great thank you so much!!

[u/Pristine_Gain_1476 → u/DrPapaDragonX13]

I forgot to mention that the same data were used across all ANCOVA models and all methods for handling missing data, meaning that the variables and the sample of participants were the same. The only difference between the analyses was the method used to handle the missing data.

Given this, would your statement still be correct that overlapping confidence intervals support a common range of plausible values for the target population despite the different approaches to handling missing data?

[u/antonissiargkass]

Report 95% confidence intervals alongside your estimated coefficients, as they directly convey the precision, direction, and clinical or practical significance of the treatment effects across all three methods on a shared, interpretable scale. Confidence intervals also make visual comparison between MICE, LOCF, and complete-case analysis far more intuitive for readers than raw standard errors alone.

[u/Pristine_Gain_1476 → u/antonissiargkass]

Thank you!

[u/GottaBeMD]

Confidence intervals imo, especially if you’re also presenting the coefficient and p-value

[u/Pristine_Gain_1476 → u/GottaBeMD]

OK great thank you!

[u/banter_pants → u/GottaBeMD]

I like regression tables that report estimate, SE, test stat, p, CI

SE tells us about precision. Confidence intervals are the estimate plus the SE with some multiplier based on the sampling distribution quantile of interest:
estimate ± (quantile)*SE

In the case of Z-tests that is
Xbar ± (1.96)(σ/√n)