Rank first, trust later: Using LLM agents for retail experiments
- LLMs can simulate shoppers and help data science teams prescreen options before running experiments. One good use case is ranking options, not estimating effect sizes, if the model's bias is order-preserving.
- Simulation quality depends on methodological details such as persona construction and representation, clear experimental instructions, response elicitation, and robustness checks.
- Recent evidence shows a major robustness problem: even when persona content stays the same, changing only the textual representation can shift simulated agent behavior by more than 70 percentage points.
- At this point, synthetic panels are safest as exploratory playgrounds unless results survive serious robustness audits across prompts, personas, and model choices.
Podcast-style summary by NotebookLM
We are taking a break in our augmented data science series to discuss another use case of LLMs in data science: can LLM agents help predict how an experiment will turn out? The idea is to simulate an A/B test or panel study on a population of synthetic customers, then ask whether the results can tell us anything useful about real customers. Agent-based simulations are not new, but LLMs make them easier to build, richer in language, and newly popular. We spent some time reflecting on when these simulations might be useful and when they might mislead.
Director's cut
Before any major retail holiday season, merchandising teams raise one critical question: which type and depth of promotion should be run to maximize sales for the coming season, given inventory and margin constraints? The challenge is that there's a whole family of promotions to pick from: a straight percentage discount, a dollar-off coupon, buy-one-get-one (free or at a discount), free shipping above a threshold, a bundle price, double loyalty points, and so on. Each has a different effect on who buys and by how much. To make things even more interesting, some of these options are brand new and have never been tested in market. So how does a data science team help pick the best promotion structure and depth?
When there's enough historical data, the data science team can predict the potential outcome of each option to guide this decision. When the data doesn't exist, though, the typical fallback mechanism is experimentation. However, in the pricing and promotions space, experimentation also comes with its own limitations. Potential spillover effects limit the number of options that can be run simultaneously. So the data science and merchandising team pick a couple of candidates, put them in front of customers, and wait weeks to read a clean lift and pick the best option. By the time the season arrives, some of the options will never have been tested.
There is an alternative way of tackling this question. Before deciding on the promotion structure, the data science team stands up a panel of synthetic shoppers: agents with personas, demographics, and purchase histories. These agents see every promotion variant and make purchase decisions overnight. By the morning, the data science and merchandising team have a directional read on the whole grid. Which promotions actually lift demand, which just subsidize purchases people would have made anyway, and roughly how large each effect might be.
The pitch is no longer hypothetical (but with limitations)
Over the past two years, the question of whether LLM agents can act as synthetic shoppers to simulate and predict human behavior has become popular. Hewitt et al. (2024) had GPT-4 simulate respondents for 70 real survey experiments and recovered 476 treatment effects across more than 100,000 participants. What the model predicted and what the experiments produced correlated at about 0.85. Chen et al. (2025) found a similar pattern, in which agent-based simulations correctly predicted the direction of 78% of the 276 published field experiments.
The reliability of agent-based simulations depends on getting a handful of methodological details right:
- Create personas: Customers are not created the same, so neither should the agents. Personas derived from real clickstream data or granular demographics reproduce actual behavior far more faithfully than generic prompts (Maier et al., 2025; Mansour et al., 2025; Castelo et al., 2026).
- Be explicit about the objective of the simulation: Explicitly describing the design and the randomization mechanism improves accuracy (Gui and Toubia, 2025).
- When the goal is to get a Likert scale, ask for a text response rather than a score: When asked to answer directly on a Likert scale, LLMs return narrow distributions. Eliciting free-text statements of intent and then mapping them back onto the scale through embeddings recovers about 90% of human test-retest reliability (Maier et al., 2025).
- Scale down the effect sizes: LLM agents are designed to be collaborative and helpful. Comparing simulated shoppers against 2,790 real sales transactions, Chen (2026) found that the agents kept deliberating and hesitated to walk away, unlike humans. As a result, effect sizes will generally be overstated and have to be adjusted down (Kolluri et al., 2025, Hewitt et al., 2024).
My take: Use LLMs to rank options, not to estimate effect sizes
Using agent-based models to simulate synthetic decision-makers is not new, but it previously required hard-coding the behavioral rules. LLMs change this by enabling autonomous agents. These agents now have a rich prior about how people behave: they take their instructions in language and can respond in language.
At this point, my working assumption is that these simulations are much better at ranking options than at sizing their effects (assuming the model's bias is order-preserving). These simulations could tell us that the bundle offer beats the dollar-off coupon and be roughly right.
Today, my favorite use case would be limited to prescreening: take the twenty promotion structures and use the agents to pick the top three that are worth an experiment with real customers. But as a guiding principle, I'd also test the agent on the promotions I have tested in the market first.
What I wouldn't do is use a simulated effect size in a business case or let a synthetic panel replace the live test.
Academic's take
The Director's Cut offers a useful blueprint for running overnight synthetic retail panels. Yet I find the approach rather optimistic. Synthetic retail panels can produce plausible rankings without being robust. It is dangerously easy to build a multi-agent LLM simulation that looks realistic while remaining fundamentally flawed.
The key issue is robustness in LLM simulations: it is critical, but difficult to establish in LLM behavior. In this post, I'll focus on a recent paper, Ye et al. (2026). I am also working on a project where I observe similar effects to this study, but that's for another day.
The Director's Cut correctly notes that reliability depends on "getting details right," such as persona creation. Personas induce heterogeneity, or differences between customers. Because these agents depend on textual personas, persona representation becomes a critical assumption. In parallel to my own findings, Ye et al. (2026) show severe implications of representational differences.
Broadly, the study shows that small, superficial design and representational choices that look interchangeable to a human can cascade through agent choices and distort macro-level outcomes.1 More specifically, in a 10-round simulated Prisoner's Dilemma, minor changes to an agent's persona format led to massive, equilibrium-flipping shifts in cooperation rates.2
The setup
The authors keep the content and strategic goals of the agent personas identical, but vary how the text is visually structured. They test three formats for persona description: a paragraph of plain prose (PLAIN), a descriptive bullet list (DESCRIPTIVE), and a structured key–value table (TABULAR). The agents are instructed to maximize their long-term payoffs and study the opponent for exploitable patterns.
A caveat
The authors actually change more than the format, adding/removing or rephrasing words even when those variations are substantively irrelevant to the problem. See the persona representations used in the study below.
Plain strategic persona.
Descriptive strategic persona.
Persona profile: - Maximize your long-term payoff - Study opponent for exploitable patterns - Adjust strategy to gain advantage - Stay strategic and competitive
Tabular strategic persona.
Persona | Value Goal | Maximize your long-term payoff Observation focus | Study opponent for exploitable patterns Strategy | Adjust strategy to gain advantage Style | Stay strategic and competitive
The results
In single-agent mode, when playing against fixed opponent policies, the TABULAR format made agents consistently the least cooperative, even though the persona content was unchanged. When two identically prompted LLM agents played against each other, the DESCRIPTIVE persona cooperated 76 percentage points less often than the PLAIN persona, and 73 percentage points less often than the TABULAR persona. In other words, an arbitrary formatting choice produced gaps larger than 70 percentage points. The main takeaway is that textual representation, even when substantively irrelevant to the problem, can change results when dealing with LLM agents.
The study shows that this lack of robustness is also highly model-dependent and unevenly distributed across model families. The exact same persona-format perturbation that caused a 76 pp shift in one frontier model caused a 77 pp shift in Claude Haiku 4.5, but had a negligible 1 pp effect on DeepSeek-v3. This differs somewhat from my own findings, but it still points to a useful second takeaway: robustness is a property of the specific model.
In closing, my take is slightly different
Use simulations as an exploratory playground, but unless a serious robustness audit proves otherwise, do not use them for decisions, even for ranking rather than effect estimation. If a minor formatting tweak can swing agent choices by 76%, even a directional read on whether a BOGO offer beats a dollar-off coupon might not reflect the behavior of actual customers. Instead, it might just be an artifact of using bullet points instead of prose in the system prompt. So, the Director’s assumption that the model’s bias is order-preserving seems like a strong one.
Frankly, the sensitivity of language models to language is not surprising; the challenge is learning how to use these tools while testing whether results survive small changes in prompts, personas, and model choice.
Footnotes
[1] This forms the basis of a fundamental challenge in complexity science: seemingly tiny changes at the agent level can lead to perturbations and large shifts at the system level. This is also a topic for another day.↩
[2] Ye et al. (2026) also propose a solution: TRAILS (Taxonomy for Robustness Audits In LLM Simulations), a systematic robustness audit in which the model is assessed at three levels: agent (micro), interactions (meso), and system (macro).↩
References
[1] Castelo, A., Foumani, Z. Z., Fan, A., Koay, K. Y., Malik, V., Zhu, Y., Li, H., Feghhi, M., Uliana, R., Xie, S., Zhang, Z., Martins, A. O., Zhao, M., Pelland, F., Faerman, J., LeBlanc, N., Glazer, A., McNamara, A., Wang, L., & Wu, Z. (2026). SimGym: Traffic-grounded browser agents for offline A/B testing in e-commerce. arXiv preprint arXiv:2602.01443.
[2] Chen, L. (2026). Simulated customers never walk away: Decision fidelity of LLM user simulators measured against real purchase outcomes. arXiv preprint arXiv:2606.20708.
[3] Chen, Y., Hu, Y., & Lu, Y. (2025). Predicting field experiments with large language models. arXiv preprint arXiv:2504.01167.
[4] Gui, G., & Toubia, O. (2025). The challenge of using LLMs to simulate human behavior: A causal inference perspective. arXiv preprint arXiv:2312.15524.
[5] Hewitt, L., Ashokkumar, A., Ghezae, I., & Willer, R. (2024). Predicting results of social science experiments using large language models.
[6] Kolluri, A., Wu, S., Park, J. S., & Bernstein, M. S. (2025). Finetuning LLMs for human behavior prediction in social science experiments. arXiv preprint arXiv:2509.05830.
[7] Maier, B. F., Aslak, U., Fiaschi, L., Rismal, N., Fletcher, K., Luhmann, C. C., Dow, R., Pappas, K., & Wiecki, T. V. (2025). LLMs reproduce human purchase intent via semantic similarity elicitation of Likert ratings. arXiv preprint arXiv:2510.08338.
[8] Mansour, S., Perelli, L., Mainetti, L., Davidson, G., & D’Amato, S. (2025). PAARS: Persona aligned agentic retail shoppers. arXiv preprint arXiv:2503.24228.
[9] Ye, J., Cao, L., Chen, D., & Ferrara, E. (2026). Stop drawing scientific claims from LLM social simulations without robustness audits. arXiv preprint arXiv:2605.18890.
