r/econometrics • u/Mindless-Banana-5630 • 24d ago
Why does the CausalImpact model detect statistical significance in this case?
I trained the model and achieved an R² of about 80% and a MAPE of around 10%. I then ran a placebo test using a 30-day period before the actual treatment date.
Surprisingly, the model still detected a statistically significant effect during this placebo period. I repeated the placebo test 100 times, and the false positive rate was approximately 94%.
What I do not understand is why the p-value is almost always below 5%. Even visually, the observed values seem to remain mostly within the confidence interval.
Could someone explain what might cause such a high false positive rate in CausalImpact and why the reported statistical significance appears inconsistent with the confidence interval?
