Why our confidence numbers just went down — on purpose
Thirty-four of our forty-two live recipe confidence scores were quietly overstating themselves, a textbook multiple-comparisons artifact, not a bug and not fraud. We re-measured every recipe with one honest protocol, corrected the thirty-four that didn't hold up, and left the eight that did. Here is the method, the evidence, and why we think a number you can reproduce beats one you can't.
Every live QGI recipe carries one headline number: a discrimination score, an AUC, rendered on its trend page as a percentage. “This recipe scores 78%.” That number is supposed to answer one question honestly: across historical crises, how well did this recipe separate the years before an event from the ordinary years in a country it had never seen? We just re-measured all 42 of them, found that 34 were overstating themselves, and turned every one of those numbers down. This post explains why we consider that an improvement.
How a confidence number gets flattered
Building a recipe means testing a shortlist of candidate time-windows against the historical record, three years, eight, eighteen, and keeping the one that discriminates best. That step, on its own, is ordinary and legitimate model selection. The problem was what happened next: the AUC we published was the score from the winning window, measured on the same walk that picked it.
That is the textbook shape of an optimistic estimate. Imagine testing a drug on nine separate patient cohorts and reporting the result from whichever cohort happened to respond best, without disclosing the other eight. The number you would publish is real, in the sense that some cohort genuinely did respond that well. It is also systematically too good, because it survived a filter built to find the best-looking result out of nine tries. Our recipe-fleet walk had the same shape: nine candidate windows, one argmax, one published number.
One honest protocol, the whole fleet
We re-measured every one of the 42 live recipes with a single honest protocol: leave-one-country-out cross-validation. For each recipe, we held out one country at a time, refit the recipe on everyone else, and asked how well it separated that held-out country's pre-crisis years from its quiet ones, repeated across every country in the recipe's evidence base. The candidate window is still chosen the same way as before; what changed is that the published number now comes from a walk that never saw the held-out country, rather than from the walk that picked the window.
This is not a new method invented for this exercise. It is the same protocol we had already used to correct two recipes individually: banking_and_financial_crisis, whose live number came down from 77% to 64% after we found its original evidence set skewed toward advanced economies and missing most of the major emerging-market crises, and military_coup, which nudged from 69% to 65% on the same re-measurement. Applying that protocol to the rest of the fleet was the obvious next step. This time we ran it against all 42 live recipes at once rather than one at a time.
What the honest measurement found
34 of the 42 recipes were overstating themselves. The typical drop was about 6 points of AUC; the worst was 17. Eight recipes held up with no meaningful change at all.
We did not take that at face value either. Two checks ran before anything shipped:
- Window-fragility sweep. For each of the 34, we swept every one of the nine candidate windows against the current evidence base and asked whether the originally published number was reachable anywhere. It was not, on any window, for any of the 34. Zero exceptions.
- Recipe-specificity check.If the honest protocol itself were simply biased downward, it would drag every recipe down by roughly the same amount. It did not: eight recipes reproduced their original number almost exactly under the identical method. The gap is a property of each recipe's original window-selection, not an artifact of how we are measuring it now.
The two recipes we had already corrected individually gave us a clean sanity check for the fleet-wide pass: run the same honest protocol at scale, and banking_and_financial_crisis and military_coup should land back on the numbers we had already published for them. They did, 0.638 and 0.652, to the decimal, with no fitting required to get there. That is the strongest piece of evidence we have that the fleet-wide correction is measuring the same thing, at scale, that we had already stood behind at retail.
What changed on the site
Three things changed, together, as one update rather than three separate ones.
The 34 overstated numbers came down, on every surface that shows them: the trend tiles, each recipe's detail page, and the fleet-wide Trend Tracker. Among the corrected 34, eight had an extra property worth flagging on its own: their honest AUC is not just lower, it is less stable, moving by three points or more depending on which countries happen to land in the held-out sample. For those, we widened the published confidence band rather than presenting a clean point estimate that implies more precision than the measurement actually has.
And we changed the words around the numbers. Recipe pages used to describe a recipe's key indicators as what is “driving” a trend. These are statistical associations between a country's trajectory and a historical outcome, not a demonstrated causal mechanism, and the language now says so: indicators that move with a trend, not indicators that drive it.
Why a lower number is the better number
We are not in the business of publishing the highest number a recipe can be made to produce. We are in the business of publishing the number that will still be there the next time someone checks. A confidence score you can't reproduce is not a lower-quality version of a good number; it is a different kind of thing, and it is worse than an honest number that happens to be smaller.
What is left, honestly measured, is modest. Across the corrected fleet the honest AUCs run from roughly 0.55 to 0.75: meaningfully better than a coin flip, in some cases considerably so, nowhere close to a crystal ball. We'd rather say exactly that than let a flattering percentage imply more than the recipe has earned.
We also checked the other direction before settling on this. Separately from the correction itself, we spent time trying to raise these numbers honestly: adding new indicators to recipes and re-measuring under the same protocol, across five trends. None of the five produced an honest gain. That is not proof that every recipe is at its absolute ceiling forever, but it is evidence that the honest numbers we are publishing now are not the result of giving up early on a higher, true number. We tried the legitimate way up, and it did not move. Telling the truth about where that leaves us is the more useful thing to publish than pretending otherwise.
What didn't change
The recipes themselves are the same: the same key indicators, the same aggregation logic, the same historical corpus each one draws on. The validation gates every recipe has to clear before it ships are the same, if anything more strictly enforced now that we have run this audit. Country coverage did not change. What changed is the number attached to each recipe, and the words we use to describe what that number means. We would rather ship 42 numbers we can defend than 42 numbers that look better and do not survive a second look.
The corrected numbers are live now across every trend page and on the Trend Tracker, which will keep tracking every recipe's status the same honest way going forward.