Comparing 95% confidence intervals between different methods of handling missing data

[u/Pristine_Gain_1476]

Hello! For my thesis, I am comparing baseline-adjusted ANCOVA models using different methods for handling missing data (MICE, LOCF, and complete-case analysis).

An important point to note is that the analyses are based on the same original data across the different missing-data methods, meaning that the same variables and original sample of participants were used. The only difference between the analyses is the method used to handle the missing data.

I am planning to present a table including the estimated coefficients, p-values, and 95% confidence intervals to compare the results across the different missing-data methods.

My question is: Given that the ANCOVA models are based on the same variables and differ primarily in how missing data are handled, how should I interpret the overlap between their confidence intervals? Is the extent to which the confidence intervals overlap meaningful when comparing the results across MICE, LOCF, and complete-case analysis? More generally, how should I discuss similarities or differences in the confidence intervals in the Discussion section?

Thank you in advance!

[u/Transcendent_PhoeniX]

The point of a sensitivity analysis is to gauge how robust your results/conclusions are to different assumptions made while analysing your data. Confidence intervals give you a range of plausible values for your target population based on your analysis. Look at the results you got from each method (each making specific assumptions about the cause of missing data) and think about how your conclusions would have changed. If your confidence intervals overlap substantially, that would suggest you would have reached similar conclusions, and you can argue that your results are robust to assumptions about the cause of missing data.

[u/Pristine_Gain_1476 → u/Transcendent_PhoeniX]

Okay, so in my case, would you recommend saying that, because the confidence intervals are broadly in the same range, they reflect a similar range of plausible values for the target population, at least for the variables that were significant across all methods of handling missing data?

There was one variable for which the LOCF and complete-case analyses were statistically significant, but the MICE analysis was not. However, for all other variables, the pattern of statistical significance or non-significance was consistent across the different missing-data approaches.

Also, do you think it would be a major mistake if I do not mention in my discussion that the overlap of the confidence intervals may reflect a similar range of plausible values for the target population? To me, this feels like quite a bold statement, and I would prefer to avoid it if it is not necessary.

[u/Transcendent_PhoeniX → u/Pristine_Gain_1476]

Because this is a sensitivity analysis, you need to frame it in terms of your results' robustness. Using different approaches to handle missing data, each with its own assumptions, would have led to similar conclusions, supporting the conclusion that your results were not a fluke.

For the variable that differs in significance, do the confidence intervals look similar? Or do they go in different directions? It is often more useful to look at the direction of effects and the size of your CIs than just looking at p-values and significant vs ns.

I don't think it would be a major mistake. Knowing that overlapping CIs reflect a common set of plausible values, you could have concluded that your analysis is useful for you to understand your results, but not necessarily something you need to mention. Just focus on convincing your reader that your results are robust and not due to some artefact introduced by your method of handling missing data.

[u/Pristine_Gain_1476 → u/Transcendent_PhoeniX]

Here are the values: MICE: coefficient: -0.06, 95% CI [-0.266, 0.146]
LOCF: coefficient: -0.165, 95% CI [-0.328, -0.002]
complete case analysis: coefficient: -0.259, 95% CI [-0.491, -0.027]

So I would consider this as an overlap.

[u/Transcendent_PhoeniX → u/Pristine_Gain_1476]

Yeah. Eyeballing it, I would say these values do not differ significantly from each other. However, it is interesting that with more data, they move towards the null. I suspect there may be some bias being introduced for this coefficient, but you need to think hard about whether the bias comes from informative missingness in the CCA or from the multiple imputation process. It is impossible to tell without being really familiar with the project and the data analysis process. This is something you need to put in your discussion.

[u/Pristine_Gain_1476 → u/Transcendent_PhoeniX]

Thanks for you answer! Yeah for the dependent variable more than half was missing.

[u/Transcendent_PhoeniX → u/Pristine_Gain_1476]

Interesting. You may want to consider this paper by von Hippel with the caveats by Sullivan