When a language model can stand in for a human in an A/B test
Running experiments on language models instead of people only works when the model output is calibrated against real human outcomes. A recent Spotify paper sets out the exact conditions — and shows why a surrogate can be falsified on past experiments but never verified for a genuinely new one.
The pitch is easy to like. Instead of running an experiment on people — slow, expensive, gated by how many users you can recruit — you run it on a panel of language models. Generate model reactions to variant A and variant B, measure the gap, ship the winner. You can test a hundred headlines before lunch. The appeal is real, and so is the failure mode, and most teams reaching for this have no principled way to tell the two apart.
A recent paper from Spotify's experimentation group, Statistical Foundations of LLM-based A/B Testing by Joel Persson, Mårten Schultzberg, and Sebastian Ankargren, gives the cleanest treatment I have seen of the only question that matters here: when does a treatment effect measured on model outputs equal the effect you would have measured on actual people?
The bar most people set is too high
There is an obvious sufficient condition. If the distribution of LLM outputs matched the distribution of human outcomes, any standard estimator you ran on the model would be valid by construction. This is also useless. LLM outputs are not human behavior; click rates, dwell times, and survey responses do not fall out of a decoder. Distributional equivalence almost never holds, and a method that requires it is a method you can never actually use.
The contribution of the paper is to replace that one strong requirement with two weaker ones, and to show they are jointly weaker than equivalence — they can hold even when the distributions plainly differ.
Two conditions, and a bridge
The first condition is surrogacy: the treatment moves the human outcome only through the model outcome. Formally, the human outcome is independent of treatment given the LLM surrogate and covariates — the model output fully mediates the effect. If a headline changes how people behave for any reason the model output does not capture, surrogacy fails.
The second is comparability: the relationship between model output and human outcome is stable across the two populations — the historical experiment you calibrate on, and the artificial sample of model runs you want to evaluate. It also requires overlap, that the model-generated outputs live in a region where you actually have human data to anchor them.
Given those two, the bridge is calibration. You learn a function that maps a model output, together with covariates, to the expected human outcome, fit on data where you observed both — a past experiment with real human results. Then you apply that learned map to fresh model outputs and read off the treatment effect as the difference in calibrated predictions between arms. You are never trusting the raw model number; you are trusting a model number that has been regressed onto human reality.
The asymmetry that should change how you use this
Here is the sentence I keep coming back to: the validity of LLM outcomes as surrogates "can only be falsified for past treatments and never verified for new ones."
Calibration and the surrogacy check both need human outcomes to compare against. On historical experiments you have them, so you can run a falsification test — does the calibrated prediction reproduce the human effect on treatments you have already run? If not, surrogacy is broken and you know it. But for a genuinely novel intervention — a new layout, a new tone, a feature nobody has shipped — there is, by definition, no human outcome to check against. The model might be a faithful surrogate or wildly off, and you cannot tell from inside the model. This is why the authors are blunt that human experiments remain indispensable for new interventions. LLM A/B testing is a tool for amortizing and extending human experiments, not for eliminating them.
Stochasticity is measurement error, and it is fixable
A language model at nonzero temperature gives a different answer each time you ask. The paper treats that as classical measurement error, which has two consequences. It inflates variance, so your effective sample size shrinks. And under linear calibration it produces attenuation bias: the estimated effect is pulled toward zero by a reliability ratio — the share of total variance that is real signal rather than sampling noise.
The fix is mechanical and it works: draw the model K times per unit and average. Averaging cuts the noise variance by a factor of K, which both shrinks the variance inflation and pushes back the attenuation — the reliability ratio climbs toward one as K grows, with most of the benefit arriving in the first few draws. Temperature is the knob on the other side: higher temperature buys output diversity at the cost of more per-draw noise, which means you need more draws to reach the same precision.
What it looks like on real data
The authors test this on the Upworthy archive — 417 A/B tests contrasting question-phrased headlines against non-question ones. The human treatment effect is small and negative, about -0.00116 on click rate. The raw LLM estimate at K=10 comes in around -0.00045: the right sign, but badly attenuated, recovering less than half of the true effect. That is exactly the conclusion you would ship if you trusted the model directly.
Calibration closes the gap. Fitting the map with random forests or gradient-boosted trees recovered the human effect within sampling error — the residual differences were not statistically distinguishable from the true value. And the draw count mattered as predicted: going from one draw to five was enough to pull the estimate within two standard errors of the human effect, with further draws adding stability.
The variables you actually control
Three design choices decide whether any of this holds. Model choice: a model trained on your product, with past treatments encoded in its training, has a better shot at surrogacy than a generic foundation model, because new interventions that resemble old ones are more likely to be mediated faithfully. Prompt: model outputs are notoriously sensitive to wording, which means prompt design is not cosmetic — it changes whether your assumptions hold, and prompt selection belongs inside the experimental workflow, not as an afterthought before it. Temperature: the precision-versus-diversity tradeoff above, which sets how many draws you need.
What I take from it
The honest framing is the useful one. An LLM surrogate is an instrument that fails loudly on the past and stays silent about the future. Calibrate it against real experiments, check it on history, average away its noise, and it can make your experimentation genuinely cheaper and faster. Skip the calibration and accept the silence — substitute the model for the user and read the raw number — and you are running experiments on a system whose only honest answer is that it cannot tell you whether the result generalizes. The discipline this paper imposes is the whole difference between the two.