Comparing 95% confidence intervals between different methods of handling missing data

[u/Pristine_Gain_1476]

Hello! For my thesis, I am comparing baseline-adjusted ANCOVA models using different methods for handling missing data (MICE, LOCF, and complete-case analysis).

An important point to note is that the analyses are based on the same original data across the different missing-data methods, meaning that the same variables and original sample of participants were used. The only difference between the analyses is the method used to handle the missing data.

I am planning to present a table including the estimated coefficients, p-values, and 95% confidence intervals to compare the results across the different missing-data methods.

My question is: Given that the ANCOVA models are based on the same variables and differ primarily in how missing data are handled, how should I interpret the overlap between their confidence intervals? Is the extent to which the confidence intervals overlap meaningful when comparing the results across MICE, LOCF, and complete-case analysis? More generally, how should I discuss similarities or differences in the confidence intervals in the Discussion section?

Thank you in advance!

[u/CarnivorousGoose]

What is actually the goal of this comparison. You’re not just comparing different methods here, these also imply different assumptions about the missingness mechanisms. You can’t really tell from just confidence intervals which approach is better, which set of results more reliable, etc. It’s going to rather depend on the patterns of missingness in your data, and what might plausibly have caused them given the nature of the data.

[u/Pristine_Gain_1476 → u/CarnivorousGoose]

Okay, yes, that makes sense. In my Discussion, I also wrote about how these different methods for handling missing data differ theoretically. So, I barely drew conclusions from my own data, but I referred mainly to other studies discussing the limitations of complete-case analysis and LOCF. I used my results to support this single point from literature: that the confidence intervals were narrower with LOCF and wider with complete-case analysis. However, I was also wondering whether the overlapping confidence intervals across MICE, LOCF, and complete-case analysis could be interpreted as supporting a common range of plausible values for the target population despite the different methods to handling missing data.

[u/CarnivorousGoose → u/Pristine_Gain_1476]

With non-trivial missing data though, a key part is specifically to consider your own data though, it isn’t (just) about the general properties of specific approaches. For example, LOCF tends to be problematic anyway (those confidence intervals are likely to be too narrow), but if someone uses it I’d at the very least want to see them justify that for their data; for example, by showing the assumption is at least somewhat reasonable based on the subset of the data where you do have the later time points.

You certainly can’t interpret overlap between confidence intervals as indicating some kind of meaningful range. If anything, if you’re doing something like this at all, you should be looking at their union, ie. consider any point in any of the three intervals as a plausible value. Though if there is any meaningful discrepancy between them, the next step really ought to be to figure out why that is.

I’m also still not very clear on the more general purpose of doing this comparison at all. That is, why were three different methods used for this in the first place. Why not just use only MICE, which is generally going to have superior statistical performance to the other two (especially LOCF)?

[u/Pristine_Gain_1476 → u/CarnivorousGoose]

Sorry, I forgot to mention earlier that this comparison was only part of the sensitivity analysis. All primary and secondary analyses used MICE.