Forecast Vintages: Why Backtests Need a Decision Time
The last forecast before delivery is not the forecast you had when you decided. ERCOT demand vintages show a 653 MW gap from using later information.
July 9, 20266 min read
Revised October 19, 2026. Original publication date retained.
A forecast vintage is the forecast as it stood at one moment in time. Backtests go wrong when they pick the latest vintage before delivery instead of the latest vintage that existed when the decision was made.
At 8 a.m. Central on September 17, 2026, an analyst using the archived ERCOT supply-and-demand feed could have selected a 7:55 a.m. publication showing 83,168 MW for the 6 p.m. target. The 5:55 p.m. publication for the same target showed 82,515 MW, a 653 MW revision. Both preceded the target, but only one preceded the decision. Taking the latest publication hands the morning’s analyst nearly ten hours of hindsight.
01
Four timestamps, four different questions
Target time is the delivery point or period being forecast. Source time is the publisher’s vintage. Availability is when the consuming system actually received the record, and decision time is the information cutoff the comparison tries to reconstruct.
The archive records the 7:55 a.m. forecast as ingested at about 7:58:20 a.m. Central, so it was available at 8 a.m. The 8 a.m. publication wasn’t ingested until about 8:03:20, and treating it as available at the exact decision assumes zero delivery latency.
Ingestion is a system-specific availability boundary, not the first moment every market participant could reach ERCOT’s publication. Studying someone else’s information set needs that party’s access record.
| Publication, Sept. 17 CDT | Recorded ingestion, CDT | Forecast for 18:00 | Eligible at 08:00? |
|---|---|---|---|
| 07:55 | 07:58:20 | 83,168 MW | Yes |
| 08:00 | 08:03:20 | 83,168 MW | No |
| 09:00 | 09:03:20 | 83,292 MW | No |
| 17:55 | 17:58:20 | 82,515 MW | No |
02
Select the eligible vintage before calculating anything
The query asks one bounded question: which forecast for the 6 p.m. target had been both published and received by 8 a.m.? The target is 23:00 UTC and the cutoff 13:00 UTC. FINAL applies the table’s revision contract and doesn’t replace vintage selection.
A lead-time study can also require, say, at least twelve hours between publication and target. Apply that condition before ranking eligible vintages, or the ranking can pick a forecast that answers an easier, shorter-horizon question.
SELECT event_time, source_time, demand_mw, ingested_at
FROM ercot.supply_demand FINAL
WHERE event_time = '2026-09-17 23:00:00'
AND series = 'intraday_forecast'
AND source_time <= '2026-09-17 13:00:00'
AND ingested_at <= '2026-09-17 13:00:00'
ORDER BY source_time DESC
LIMIT 103
A later observation doesn’t fix a mismatched target
Realized demand at 6 p.m. is 81,572 MW. Subtracting it from the two forecasts gives apparent misses of 1,596 MW and 943 MW, so a latest-before-delivery comparison looks 653 MW closer using information the morning analyst never had.
That subtraction demonstrates vintage selection and isn’t a forecast-skill score. In this feed the on-the-hour intraday values are hourly averages with interpolated points between them, while the realized row is an instantaneous snapshot. A valid accuracy test first matches the forecast’s delivery-period statistic to the observation.
The distinction matters twice: preserve which forecast existed, then preserve what it meant. Correct vintage selection can’t rescue a comparison of an hourly average against a point in time.
04
Public disclosure is not operator visibility
ERCOT’s 60-day SCED disclosure gives detailed retrospective resource records. Its release delay governs when a public researcher can use them, and says nothing about what ERCOT or a participant knew from operational systems during dispatch.
The same holds for a revised plant registration or queue snapshot: a later record improves today’s account of history and is still ineligible for reconstructing yesterday’s decision.
When the question needs both, store both versions: the best current account of what happened, and the evidence available at the chosen decision time.