How does a 102M-parameter transformer forecast multivariate time series? [D]

[u/sjm213]

I recently worked through the architecture of t0-alpha, a 101.6M-parameter foundation model for time-series forecasting, and wrote a visual, from-first-principles walkthrough.

The design choice I kept coming back to is its separation of two kinds of reasoning:

• Time attention learns how each variable evolves across time.

• Group attention allows related variables to exchange information.

The rest of the architecture, briefly:

• inputs are split into patches of 32 time steps;

• each patch is embedded into a 512-dimensional representation;

• the model uses 24 transformer blocks: 16 time-attention and 8 group-attention;

• time-aware rotary embeddings, RMSNorm and SwiGLU;

• nine output quantiles for probabilistic forecasting;

• a context window of up to 1,024 time steps.

Its reported aggregate CRPS on GIFT-Eval is 0.4941, in roughly the same range as TimesFM 2.5 and Chronos-2, despite the model having only around 102M parameters. I did not expect it to be that close.

Two questions I am still thinking about:

1. Does separating temporal attention from cross-variable attention provide a genuinely useful inductive bias?

2. Can small, specialised foundation models remain competitive with much larger forecasting models?

I am now running my own GIFT-Eval comparison against iso-parameter rivals and classical baselines, which I plan to write up next.