Benchmarks · ensemble 0.7.5
Level with the best forecasting models, under an open licence.
The Ephemeris ensemble blends six open-weights foundation models, weighting each by its measured accuracy at the requested horizon. Scored with each benchmark's own harness, it is level with the top of TIME. On GIFT-Eval it scores better than every model it blends and than the larger Toto-2.0 2.5B; a few leaderboard entries score better, including TimesFM-3, which cannot be used commercially.
31 models
TIME
Lower is better; MASE and CRPS are ratios to seasonal naive. The ensemble has the best average MASE rank of all 31 models (5.684), and is within 0.002 of the top on both scores.
| Model | MASE | CRPS | Note |
|---|---|---|---|
| QiYao-M | 0.638 | 0.536 | |
| Ephemeris ensemble | 0.639 | 0.538 | |
| TimesFM-3 | 0.640 | 0.536 | non-commercial licence |
| Toto-2.0 2.5B | 0.642 | 0.539 | |
| Toto-2.0 313M | 0.644 | 0.542 | |
| Chronos-2 | 0.662 | 0.556 |
97 configurations
GIFT-Eval
Lower is better. MASE and CRPS are ratios to seasonal naive, geometric mean over the 97 configurations; MASE_Rank and CRPS_Rank are the mean rank per configuration among the leaderboard's 140 entries. Published rows are copied from the official leaderboard in its default order (MASE_Rank) and none is flagged for test-data leakage. The ensemble row is our own run on the same aggregation, not a leaderboard submission, so it has no rank.
| Model | MASE | MASE_Rank | CRPS | CRPS_Rank | Note |
|---|---|---|---|---|---|
| TimesFM-3 | 0.6668 | 31.299 | 0.4557 | 29.381 | zero-shot; non-commercial licence |
| Toto-2.0-2.5B | 0.6956 | 49.237 | 0.4759 | 49.649 | pretrained |
| Chronos-2 | 0.6978 | 52.216 | 0.4854 | 53.000 | pretrained |
| TiRex-2-Zeroshot | 0.6973 | 57.402 | 0.4781 | 51.856 | zero-shot |
| Ephemeris ensemble | 0.6841 | — | 0.4662 | — | our run, not a leaderboard submission |
Source: GIFT-Eval leaderboard, read 2026-10-07 (shown there to three decimals; four decimals from the per-configuration results files), and the GIFT-Eval paper.
Why an ensemble
Ensemble vs its best single model
How much worse the best individual model in the panel is than the ensemble, per benchmark. A different model is best on each, which is why no single model is the right default. More in Does ensembling help? and the model-by-model comparison.
| Benchmark | Configurations | Best single model | Its CRPS vs ensemble | Its MASE vs ensemble |
|---|---|---|---|---|
| GIFT-Eval | 97 | Chronos-2 | +3.6% | +2.9% |
| TIME | 134 | Toto 2 | +1.6% | +2.5% |
| fev-bench | 96 | TimesFM 2.5 | +1.1% | +1.9% |
| BOOM | 1,073 | Toto 2 | +0.0% | -0.2% |
Reproducibility
Method
Scores
CRPS (probabilistic accuracy, from nine quantiles) and MASE (point accuracy of the median), each divided by a seasonal-naive forecast of the same window and averaged geometrically across configurations, as the benchmarks themselves report them.
Harnesses
TIME: the benchmark's own runner and compute_local_leaderboard.py, ranked against its published model outputs. GIFT-Eval: the official harness and published seasonal naive. These are our runs, not leaderboard submissions. The ensemble weights were fixed on separate data before any of these benchmarks were scored.
Same ensemble, one request