Skip to main content

Transformer networks for generic time series prediction

2024-06-17 — 2026-10-20

quality 4.4

In Which Timestral Is Deployed With a 209M Parameter Architecture, Horizons Exceeding 1,024 Steps Are Serialized as Median Blocks, and Competitors Like Chronos-2 and TimesFM-3 Are Challenged on the GIFT-Eval Benchmark.

language
machine learning
meta learning
neural nets
NLP
stringology
time series
Figure 1

Transformers for generic time series prediction.

For a while it wasn’t obvious that transformers were any good at this, which is kind of amazing since transformers can notoriously predict that most recondite of time series, human language. Nonetheless, Zeng et al. (2023) found that a single linear layer beat the transformer forecasters of the day on the standard long-horizon benchmarks.

Anyway, now we are in the future and it works. We can, in fact, pretrain one model on a large heap of heterogeneous series, then forecast new series in-context. TimesFM (Das et al. 2024), TimeGPT (Garza, Challu, and Mergenthaler-Canseco 2024), and Time-MoE (Shi et al. 2024) are early examples. The designs have converged on a few shared ingredients:

1 Timestral

Timestral is Mistral’s entry (blog post, code). It runs on GPU or CPU.

Horizons past 1,024 steps are forecast block by block and use only the median forecast from the previous block, appending each block’s median forecast to the context. So beyond the first block the quantiles are conditional on a single median path, rather than integrating over uncertainty in the earlier blocks; I would expect the far-horizon intervals to be too narrow?

The model card claims first place on GIFT-Eval, fev-bench, and TIME, ahead of t0-gamma, TimesFM-3, Toto-2.0, and Chronos-2, albeit by a probably-not-significant 1% margin.

2 Others

  • Chronos (Amazon): the original scales and quantizes values into a fixed vocabulary of tokens and trains an off-the-shelf T5 language model on them with cross-entropy loss (Ansari et al. 2024), i.e., it treats a time series literally as a language. Chronos-2 is a 120M-parameter encoder-only successor which does univariate, multivariate, and covariate-informed forecasting (Ansari et al. 2025).
  • TimesFM (Google): decoder-only, now at TimesFM-3, which adds multivariate forecasting.
  • Toto (Datadog): trained heavily on observability metrics (Cohen et al. 2025). Toto 2.0 spans 4M to 2.5B parameters and is pitched as the first time series foundation model that gets reliably better with scale (Khwaja et al. 2026).

3 Benchmarks

See time series forecasting benchmarks.

4 References

Ansari, Shchur, Küken, et al. 2025. “Chronos-2: From Univariate to Universal Forecasting.”
Ansari, Stella, Turkmen, et al. 2024. “Chronos: Learning the Language of Time Series.”
Cai, Zhu, Wang, et al. 2024. “MambaTS: Improved Selective State Space Models for Long-Term Time Series Forecasting.”
Cohen, Khwaja, Doubli, et al. 2025. “This Time Is Different: An Observability Perspective on Time Series Foundation Models.”
Das, Kong, Sen, et al. 2024. “A Decoder-Only Foundation Model for Time-Series Forecasting.” In.
Garza, Challu, and Mergenthaler-Canseco. 2024. “TimeGPT-1.”
He, Yang, Cheng, et al. 2025. “Chaos Meets Attention: Transformers for Large-Scale Dynamical Prediction.”
Khwaja, Lettieri, Woo, et al. 2026. “Toto 2.0: Time Series Forecasting Enters the Scaling Era.”
Nguyen, Brandstetter, Kapoor, et al. 2023. “ClimaX: A Foundation Model for Weather and Climate.”
Nishikawa, and Suzuki. 2024. “State Space Models Are Comparable to Transformers in Estimating Functions with Dynamic Smoothness.”
Shi, Wang, Nie, et al. 2024. “Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts.” In.
Zeng, Chen, Zhang, et al. 2023. “Are Transformers Effective for Time Series Forecasting?” In Proceedings of the AAAI Conference on Artificial Intelligence.