11 Jun 2026 · 5 min read

Can we trust a SHAP value?

What six benchmarks and a few thousand near-optimal models say about explanation stability.

Question

SHAP takes a single prediction and splits it into a share for each feature: how much the most recent reading pushed a forecast up, how much the hour of day pulled it down. It gives one such explanation per model. But for most forecasting problems there are lots of models that score about the same. The Rashomon set is the name for that crowd: every model that fits nearly as well as the best one. They agree on the error and not much else.

So I wanted to know one thing. If two models forecast about equally well, do they tell the same story about why? And does that story survive when the model is refit on the next slice of time? This is from my MSc thesis: six forecasting benchmarks run through two AutoML systems.

Setup

I built the Rashomon set like this: take the best model by MAE, then keep every model within a factor of (1 + ε) of it. I did that at five tolerances, ε in {0.02, 0.05, 0.10, 0.20, 0.30}.

Then I built that set two ways, because the way we search for models might change the answer. AutoGluon on best_quality, which bags and stacks its models. And H2O AutoML, which runs a grid over gradient-boosted trees, random forests, extremely randomised trees, and a linear model. I ran this on six forecasting benchmarks, including the ETT family and M4 Monthly, three seeds each, with rolling-origin splits so nothing leaks back from the future.

For each set I computed SHAP importances and compared the feature rankings with Spearman ρ, which runs from -1 to 1: at 1 two rankings are identical, at 0 they have nothing to do with each other. I compared them two ways. Inside a single fit, whether the equally-good models agree with each other. And across time, whether the ranking survives when I refit the set on a later slice of history.

Mean Spearman ρ at ε = 0.10 across six forecasting benchmarks and two AutoML frameworks. The AutoGluon column stays dark and stable. The H2O column lightens sharply for Electricity and ETTm1.

Within a single fit

Inside any one training window, the crowd mostly agrees. Pick a fit and the equally-good models rank the features about the same way. Even on Electricity under H2O, where the set had around a dozen models and which gave me the most trouble later, the mean pairwise Spearman ρ inside a set read 0.930. The worry I started with, that swapping one near-optimal model for another rewrites the SHAP plot, mostly did not happen inside a fit.

Under AutoGluon there was often nothing to swap. On four of the six benchmarks the AutoGluon set was a single model at the tight tolerances, so there was no crowd to disagree with. And on ETTh2, where AutoGluon did keep a random forest and an extra-trees model next to its boosted trees, the ranking stayed stable anyway (ρ 0.853). Its steadiness is partly built in, from small sets that rarely reach for a model that would disagree.

Across the time windows

Refitting on later history changed the picture. Under AutoGluon the ranking stayed put: temporal ρ read between 0.774 (Cable Demand) and 0.957 (Electricity), and it barely moved as I widened ε. Under H2O, four of the six benchmarks stayed in that same range, ρ around 0.76 to 0.93. Two did not. Electricity fell to ρ = 0.091, and ETTm1 to 0.485.

It took me a while to see why, because within any single fit those same H2O models had agreed. What got me is that the Electricity collapse wasn’t gradual. At ε = 0.02 the ranking was stable from one window to the next, ρ around 0.97. One step wider, at ε = 0.05, it fell to around 0.09. That step is where an extremely randomised trees model (XRT) first becomes accurate enough to qualify for the set, and in the windows where it appears, the averaged ranking lurches.

Then I looked at what XRT actually did, and part of what I found came from the explainer itself. On one Electricity fit, the best model in the set put target_lag1 first, the most recent reading. An equally accurate model put target_lag5 first, the value five steps back. Every model whose SHAP values were computed exactly agreed on target_lag1. The one that disagreed was XRT, and H2O explains XRT with a fast approximation (Saabas) that does not obey the same accounting rules. Here it placed its SHAP values on a scale thousands of times larger than the exact ones. When I restricted the comparison to the models with exact SHAP, the within-fit agreement on Electricity came back up by about 0.09. So part of the instability came from the SHAP method itself.

Mean Spearman ρ (±1σ) against ε, one panel per benchmark, AutoGluon against H2O. AutoGluon stays flat across every ε. H2O falls in a single sharp step on the two benchmarks where an extra-trees model qualifies for the near-optimal set.

What it means for trusting explanations

The strong version of this, that the model search decides whether an explanation is trustworthy, is more than six benchmarks can carry. Ten of the twelve dataset-and-framework pairs were stable across time. The two that came apart were both under H2O, and both arrived with the same thing: an extra-trees model, explained by approximation, qualifying for the set. That is the narrower claim, and it is the one the evidence supports.

It matters when someone has to act on the chart. If the top driver of an electricity-load forecast flips between the most recent reading and one five steps back depending on which equally-good model, or which week we refit on, a single importance chart is not enough to sign off on.

The fix is cheap, so it is worth doing. Refit a few of the near-optimal models, and refit on the next window too, then recompute the SHAP importances and see whether the ranking survives before it goes in a slide deck. If it moves, show the spread instead of one model’s bar chart. Trust the explanation once we have watched it survive being swapped for an equally good model.

The full thesis, the six benchmarks, and the code are in the project this note hangs off: SHAP Stability Across the Rashomon Set.