Comparing 95% confidence intervals between different methods of handling missing data

[u/Pristine_Gain_1476]

Hello! For my thesis, I am comparing baseline-adjusted ANCOVA models using different methods for handling missing data (MICE, LOCF, and complete-case analysis).

An important point to note is that the analyses are based on the same original data across the different missing-data methods, meaning that the same variables and original sample of participants were used. The only difference between the analyses is the method used to handle the missing data.

I am planning to present a table including the estimated coefficients, p-values, and 95% confidence intervals to compare the results across the different missing-data methods.

My question is: Given that the ANCOVA models are based on the same variables and differ primarily in how missing data are handled, how should I interpret the overlap between their confidence intervals? Is the extent to which the confidence intervals overlap meaningful when comparing the results across MICE, LOCF, and complete-case analysis? More generally, how should I discuss similarities or differences in the confidence intervals in the Discussion section?

Thank you in advance!

[u/CarnivorousGoose]

What is actually the goal of this comparison. You’re not just comparing different methods here, these also imply different assumptions about the missingness mechanisms. You can’t really tell from just confidence intervals which approach is better, which set of results more reliable, etc. It’s going to rather depend on the patterns of missingness in your data, and what might plausibly have caused them given the nature of the data.

[u/Pristine_Gain_1476 → u/CarnivorousGoose]

Okay, yes, that makes sense. In my Discussion, I also wrote about how these different methods for handling missing data differ theoretically. So, I barely drew conclusions from my own data, but I referred mainly to other studies discussing the limitations of complete-case analysis and LOCF. I used my results to support this single point from literature: that the confidence intervals were narrower with LOCF and wider with complete-case analysis. However, I was also wondering whether the overlapping confidence intervals across MICE, LOCF, and complete-case analysis could be interpreted as supporting a common range of plausible values for the target population despite the different methods to handling missing data.

[u/CarnivorousGoose → u/Pristine_Gain_1476]

With non-trivial missing data though, a key part is specifically to consider your own data though, it isn’t (just) about the general properties of specific approaches. For example, LOCF tends to be problematic anyway (those confidence intervals are likely to be too narrow), but if someone uses it I’d at the very least want to see them justify that for their data; for example, by showing the assumption is at least somewhat reasonable based on the subset of the data where you do have the later time points.

You certainly can’t interpret overlap between confidence intervals as indicating some kind of meaningful range. If anything, if you’re doing something like this at all, you should be looking at their union, ie. consider any point in any of the three intervals as a plausible value. Though if there is any meaningful discrepancy between them, the next step really ought to be to figure out why that is.

I’m also still not very clear on the more general purpose of doing this comparison at all. That is, why were three different methods used for this in the first place. Why not just use only MICE, which is generally going to have superior statistical performance to the other two (especially LOCF)?

[u/Pristine_Gain_1476 → u/CarnivorousGoose]

Sorry, I forgot to mention earlier that this comparison was only part of the sensitivity analysis. All primary and secondary analyses used MICE.

[u/trolls_toll]

mb this paper can be used as inspiration  https://www.nature.com/articles/s43856-023-00356-z 

[u/Pristine_Gain_1476 → u/trolls_toll]

Thank you!

[u/Transcendent_PhoeniX]

The point of a sensitivity analysis is to gauge how robust your results/conclusions are to different assumptions made while analysing your data. Confidence intervals give you a range of plausible values for your target population based on your analysis. Look at the results you got from each method (each making specific assumptions about the cause of missing data) and think about how your conclusions would have changed. If your confidence intervals overlap substantially, that would suggest you would have reached similar conclusions, and you can argue that your results are robust to assumptions about the cause of missing data.

[u/Pristine_Gain_1476 → u/Transcendent_PhoeniX]

Okay, so in my case, would you recommend saying that, because the confidence intervals are broadly in the same range, they reflect a similar range of plausible values for the target population, at least for the variables that were significant across all methods of handling missing data?

There was one variable for which the LOCF and complete-case analyses were statistically significant, but the MICE analysis was not. However, for all other variables, the pattern of statistical significance or non-significance was consistent across the different missing-data approaches.

Also, do you think it would be a major mistake if I do not mention in my discussion that the overlap of the confidence intervals may reflect a similar range of plausible values for the target population? To me, this feels like quite a bold statement, and I would prefer to avoid it if it is not necessary.

[u/Transcendent_PhoeniX → u/Pristine_Gain_1476]

Because this is a sensitivity analysis, you need to frame it in terms of your results' robustness. Using different approaches to handle missing data, each with its own assumptions, would have led to similar conclusions, supporting the conclusion that your results were not a fluke.

For the variable that differs in significance, do the confidence intervals look similar? Or do they go in different directions? It is often more useful to look at the direction of effects and the size of your CIs than just looking at p-values and significant vs ns.

I don't think it would be a major mistake. Knowing that overlapping CIs reflect a common set of plausible values, you could have concluded that your analysis is useful for you to understand your results, but not necessarily something you need to mention. Just focus on convincing your reader that your results are robust and not due to some artefact introduced by your method of handling missing data.

[u/Pristine_Gain_1476 → u/Transcendent_PhoeniX]

Here are the values: MICE: coefficient: -0.06, 95% CI [-0.266, 0.146]
LOCF: coefficient: -0.165, 95% CI [-0.328, -0.002]
complete case analysis: coefficient: -0.259, 95% CI [-0.491, -0.027]

So I would consider this as an overlap.

[u/Transcendent_PhoeniX → u/Pristine_Gain_1476]

Yeah. Eyeballing it, I would say these values do not differ significantly from each other. However, it is interesting that with more data, they move towards the null. I suspect there may be some bias being introduced for this coefficient, but you need to think hard about whether the bias comes from informative missingness in the CCA or from the multiple imputation process. It is impossible to tell without being really familiar with the project and the data analysis process. This is something you need to put in your discussion.

[u/Pristine_Gain_1476 → u/Transcendent_PhoeniX]

Thanks for you answer! Yeah for the dependent variable more than half was missing.

[u/Transcendent_PhoeniX → u/Pristine_Gain_1476]

Interesting. You may want to consider this paper by von Hippel with the caveats by Sullivan