RealCast-bench
Time series foundation models on proprietary commercial panels
Leaderboard
Loading results…
See details
Relative score is the geometric mean over tasks of each task's error divided by Seasonal Naive's error on the same task. Lower is better, and Seasonal Naive scores 1.000. Skill is the same number read as a percentage improvement over Seasonal Naive. Win rate is the share of (task, other configuration) pairings in which a configuration has the lower error, with ties counted as half. It is computed among the configurations shown, so filtering models changes it.
MASE scores the median forecast, scaled by a seasonal-naive error on each series's own history. WQL scores the quantiles 0.1 to 0.9.
A task is one dataset at one forecast horizon, and a dataset is one panel at one entity level and one frequency. Ratios are not comparable across boards: daily Seasonal Naive repeats last week's value, a much stronger reference than the weekly and monthly Seasonal Naive, which repeat the value from a year earlier.
Covariates means the model also saw the panel's other measured series up to the forecast origin, plus calendar features. Every pretrained model runs zero-shot: none is fine-tuned on these panels, and every model runs with default settings.
Pairwise comparison
See details
Computed as on fev-bench, with its code. Win rate is the share of tasks on which Model 1 has the lower error than Model 2, ties counted as half. Skill score is 1 minus the geometric mean over tasks of Model 1's error divided by Model 2's, with ratios clipped to [0.01, 100]. 95% confidence intervals (on hover) come from 1,000 bootstrap resamples of tasks. The 15 configurations with the highest average win rate are shown, plus Seasonal Naive. These matrices follow the board and metric selected; the panel, level and model filters do not apply to them.
Results for individual tasks
Citation
If you use these results, please cite RealCast-bench: Evaluating Time Series Foundation Models on Proprietary Commercial Panels.
About
RealCast-bench evaluates time series foundation models zero-shot on proprietary commercial panels: card spend, web visits, app downloads and usage, foot traffic, job openings, digital ad spend, Amazon and TikTok Shop sales, and Rx claims. There are 10 source panels, 81 datasets and 290 tasks across a weekly, a monthly and a daily board, with a test year running from October 2025 to September 2026.
Public benchmarks are built from public data. RealCast-bench asks whether their rankings hold on the kind of brand- and company-level commercial series that forecasts are actually used for.
Resources
- Paper: RealCast-bench: Evaluating Time Series Foundation Models on Proprietary Commercial Panels
- Results: carbon-arc/RealCast-bench, the leaderboard and every per-task score behind it
The panels themselves are proprietary and are not released. Every number on this leaderboard can be recomputed from the per-task results.
Submit your model
Because the data is private, we run submissions ourselves. Open a discussion in this Space's Community tab with a link to your model's public weights and forecasting package. We run it through the same wrapper, entities and windows as every other configuration and add it to all three boards.