Four failed forecasts — what a year of trying to predict ERCOT prices actually taught us

Most firms selling energy analytics show you a backtest. A backtest is a story about a model told by the person who built it, and it is almost always the best of many attempts. You are shown the survivor and not the graveyard.

So here is our graveyard.

Attempts one, two and three: beat the day-ahead price

The premise is the most natural one in the business. ERCOT publishes a day-ahead price and a real-time price, and the gap between them is where a battery or a trading desk makes its money. Forecast that gap better than the market does and everything else follows.

We built three statistical models on a shared harness, so each was scored on identical rows against an identical baseline — ERCOT’s own day-ahead price. All three lost. Not narrowly, in a way better features might rescue, but to a number that is free, public, and available before the day starts.

That is less surprising than it first looks, and the reason matters. A day-ahead price is not a weather forecast. It is the clearing outcome of a market in which hundreds of participants, several of them very well resourced, have already committed money to their own best estimate. Beating it means beating their aggregate with less information than they collectively hold.

Attempt four: find a fundamental the market has missed

If the price itself cannot be forecast, the orthodox next move is to find a physical driver the market prices slowly. Generator outages are the obvious candidate: when capacity goes offline supply tightens, and ERCOT discloses outages in detail.

We tested it across 13.6 million records spanning 2017 to 2026. Outage capacity carries no usable price signal. The most obvious remaining fundamental, tested at scale, on the market that discloses more than any other in the United States. Nothing.

Four attempts, four failures. Taken together they do something more valuable than a fifth attempt would have: they rule out a category.

Build the scoreboard before the model

Every one of those four attempts was scored against the same baseline on the same rows. That is the only reason we can say they failed rather than say they underperformed — and it is the part most teams skip. If a fifth attempt is ever made it gets scored against that same scoreboard. Otherwise it is not an attempt, it is a demonstration.

Why publish it

Three reasons, in ascending order of how much they should matter to you.

First, it saves you the same year. If you are standing at the start of this problem with a data science team and a budget, the expected value of prediction in ERCOT is lower than it looks and the baseline you have to beat is better than it looks.

Second, it is a test of the firm. Anyone can publish a win. A firm that publishes its four most commercially attractive failures is telling you something about what its other claims are worth — because the same measurement discipline that killed these would have killed the others if they deserved it.

Third, and most useful: it forces the work somewhere better. If the price cannot be predicted, the money has to come from something that does not require predicting it — structural biases that have held their direction for years, and the measurable gap between what an asset earned and what the same published prices would have paid a perfect operator at that node. Neither needs a forecast. Both can be counted.

That is where we went next, and it is the only reason this piece is not simply an admission of defeat.

Benchmark my fleet

The measurement that replaced prediction is specific and per asset: what each battery you own actually captured, against what the same published prices would have paid a perfect operator at that node, with that asset’s own power and duration. No forecast is involved. If you would like it run on your fleet, get in touch.