Transformer networks for generic time series prediction
2024-06-17 — 2026-10-20
In Which Timestral Is Deployed With a 209M Parameter Architecture, Horizons Exceeding 1,024 Steps Are Serialized as Median Blocks, and Competitors Like Chronos-2 and TimesFM-3 Are Challenged on the GIFT-Eval Benchmark.
On this page
Transformers for generic time series prediction.
For a while it wasn’t obvious that transformers were any good at this, which is kind of amazing since transformers can notoriously predict that most recondite of time series, human language. Nonetheless, Zeng et al. (2023) found that a single linear layer beat the transformer forecasters of the day on the standard long-horizon benchmarks.
Anyway, now we are in the future and it works. We can, in fact, pretrain one model on a large heap of heterogeneous series, then forecast new series in-context. TimesFM (Das et al. 2024), TimeGPT (Garza, Challu, and Mergenthaler-Canseco 2024), and Time-MoE (Shi et al. 2024) are early examples. The designs have converged on a few shared ingredients:
- Patching: a chunk of consecutive time steps becomes one token, so a long context costs fewer attention operations, as made famous in Vision Transformers (ViTs).
- Quantile outputs rather than point forecasts, i.e., probabilistic forecasting by default.
- In the newer ones, attention across variates as well as across time, so that multivariate targets and covariates are co-equal first-class citizens in the model.
1 Timestral
Timestral is Mistral’s entry (blog post, code). It runs on GPU or CPU.
Horizons past 1,024 steps are forecast block by block and use only the median forecast from the previous block, appending each block’s median forecast to the context. So beyond the first block the quantiles are conditional on a single median path, rather than integrating over uncertainty in the earlier blocks; I would expect the far-horizon intervals to be too narrow?
The model card claims first place on GIFT-Eval, fev-bench, and TIME, ahead of t0-gamma, TimesFM-3, Toto-2.0, and Chronos-2, albeit by a probably-not-significant 1% margin.
2 Others
- Chronos (Amazon): the original scales and quantizes values into a fixed vocabulary of tokens and trains an off-the-shelf T5 language model on them with cross-entropy loss (Ansari et al. 2024), i.e., it treats a time series literally as a language. Chronos-2 is a 120M-parameter encoder-only successor which does univariate, multivariate, and covariate-informed forecasting (Ansari et al. 2025).
- TimesFM (Google): decoder-only, now at TimesFM-3, which adds multivariate forecasting.
- Toto (Datadog): trained heavily on observability metrics (Cohen et al. 2025). Toto 2.0 spans 4M to 2.5B parameters and is pitched as the first time series foundation model that gets reliably better with scale (Khwaja et al. 2026).
