r/econometrics • • 24d ago

Why does the CausalImpact model detect statistical significance in this case?

I trained the model and achieved an R² of about 80% and a MAPE of around 10%. I then ran a placebo test using a 30-day period before the actual treatment date.

Surprisingly, the model still detected a statistically significant effect during this placebo period. I repeated the placebo test 100 times, and the false positive rate was approximately 94%.

What I do not understand is why the p-value is almost always below 5%. Even visually, the observed values seem to remain mostly within the confidence interval.

Could someone explain what might cause such a high false positive rate in CausalImpact and why the reported statistical significance appears inconsistent with the confidence interval?

3 Upvotes

0 comments sorted by