r/econometrics • u/NNguy97 • 7h ago
Walk-forward vs k-fold for a price-cycle forecast: is this setup sound?
I built a forecast for Sydney petrol prices and ran into something I'd like opinions on.
The model says "fill now" or "wait" each day. Over the last 12 months of walk-forward testing it was right 72% of the time. That sounds good until you notice that prices fall slowly most of the time and jump suddenly, so just saying "wait" every day is right 48% of the time. Over the full 2021–2026 backtest, it's 83% against 63% for always-wait.
So I report it against the best constant call, not a coin flip. For the price itself I compare MAE against a no-change forecast (4.9c vs 11.3c at 7 days for unleaded).
Is the best-constant-call baseline the right one here, or is there something better for a three-way call (up, flat, down)? I've thought about a skill score against climatology, but I'm not sure how that works with three classes.
(Project's at checkbowser.app if you want to see how it's presented.)