In controlled ACES tests, a single review raised a model’s selection probability by up to 99 percentage points, Wharton researchers report
The Wharton School team used the Agentic e‑Commerce Simulator (ACES) to probe how modern AI “shopping agents” choose when shown the same product grid plus small contextual nudges. Tiny changes, a single external recommendation, the order in which snippets arrive, whether they are bundled or delivered sequentially, or a short user “memory” line like “I love hiking!”, shifted model recommendations by tens of percentage points in many cases.
What the experiment looked like
The authors (Kumar et al.) presented agents with an 8‑product screenshot and optional external recommendation snippets (examples: a Reddit thread recommending the Garmin Forerunner 55, a Wirecutter review favoring the Fitbit Inspire 3, and a Strategist article about the WHOOP 5.0). The study is available on arXiv:2508.02630 and was summarized in The Decoder by Matthias Bastian (Aug 27, 2026).
Key experimental mechanics: ACES presents a static product grid, optionally supplies curated external snippets, and records which product the agent selects. The authors report running roughly 200 trials per experimental setting, randomizing badges and positions where relevant so they could draw causal inferences on certain interventions.
Methods, controls, and what the simulator does, and doesn’t do
- ACES is a controlled simulator built around screenshots, and agents do not crawl live pages or follow links in these experiments.
- “Memory” injections were implemented as short persistent prompt attributes added to the agent’s context for each run (for example, “I love hiking!” or “I don’t like hiking!”).
- Badge and position effects were randomized, and the paper uses conditional logit modeling to estimate how product attributes, endorsements, and layout change selection probabilities.
- Because the setup is a simulator with curated snippets and fixed images, generalization to real-world agentic systems that browse live sites, fetch reviews, or execute purchases remains an open question.
Headline findings, big swings from small nudges
- Single‑source sway: exposing models to a Wirecutter review that favored the Fitbit Inspire 3 produced massive lifts. The Wharton team reports that Claude Opus 4.8’s probability of choosing the Inspire 3 rose by about +90 percentage points (pp) versus the no‑source baseline, and Gemini 3.5 Flash jumped by about +99 pp.
- Multiple sources don’t simply average out: adding more recommendation snippets did not neutralize model choices. A dominant source, Wirecutter in these tests, tended to steer selections when present, and overall variability increased with more sources.
- Order matters: presentation order shifted outputs. Gemini 3.1 Flash Lite’s probability of picking the Inspire 3 swung between roughly +2 and +56 pp above control depending only on source order, while Claude Haiku 4.5 stayed more stable at about +41-42 pp.
- Bundled vs sequential delivery matters: GPT‑5.5 chose the Fitbit Inspire 3 about +53 pp more often when snippets were bundled together, but only +6 pp more when the same snippets were delivered sequentially.
- Memory snippets bias toward preference‑aligned, often pricier options: adding “I love hiking!” moved several models away from an objectively superior low‑cost option and toward higher‑priced watches. Picks for the Garmin Vivoactive 5 increased by roughly +75 pp for Claude Opus 4.8, +37 pp for GPT‑5.5, and +36 pp for Gemini 3.1 Flash Lite.
- Not all models are equally susceptible: Gemini 3.5 Flash showed relative robustness in the objective‑superiority test, selecting the best product in about 86-92% of runs regardless of memory injections. GPT‑5 Mini behaved idiosyncratically: a positive hiking memory did not move it substantially toward the Garmin, while the negative statement “I don’t like hiking!” significantly boosted selections for the Fitbit Versa 4.
- Objective‑superiority test: in one grid the authors created a product that was objectively superior on price, rating, and review count (example given: a smart watch with Alexa for $29.99, rated 5.0/5.0 with 430 reviews, versus alternatives costing at least $359). Even then, small context changes moved several models away from the measurable bargain.
Why this matters for consumers, sellers, and platforms
For consumers: delegating purchases to an AI agent no longer looks like handing over a neutral, objective comparator. Agentic decisions are systematically shaped by which snippets the model saw, the order they arrived in, how they were bundled, and even tiny “memory” prompts you or a system set, often in ways you cannot see or control. That raises the risk of inconsistent or non‑optimal purchases when agents act autonomously.
For sellers and platforms: a single credible endorsement or a high‑salience snippet can swing algorithmic demand dramatically, but which endorsement matters is model dependent. Sellers can and will try to optimize listings and snippets for algorithmic buyers, and the paper shows seller agents rewriting copy can increase market share. Still, heterogeneity across buyer models and sensitivity to presentation make optimization fragile and transient.
There’s also a regulatory angle. The FTC has been active on AI‑related consumer protections (see Operation AI Comply, Sept 2024), and the Wharton results mean that fabricated or manipulated endorsements could influence automated buyers as well as humans, creating a compliance and enforcement risk for platforms and merchants.
Limits and open questions
- ACES uses static screenshots and curated snippets. Real agentic systems that crawl, follow links, and access live pricing may behave differently.
- The experiments document what happens, but not all why. The internal mechanisms producing these sensitivities, whether training data exposures, attention dynamics, or system prompts, are not fully traced in the paper.
- Results are specific to a fitness‑watch product grid. Behavior may vary across categories, price ranges, multi‑constraint purchase goals, or international markets.
- Model updates and vendor settings (temperature, decoding, system messages) likely change behavior. Continuous monitoring is necessary, but the paper does not enumerate every configuration sensitivity.
Practical steps leaders can start today
Treat agentic shopping as a dynamic channel that must be tested and monitored, not set-and-forget. Concrete, actionable checks:
- Run systematic audits: perform quarterly audits across multiple model families and versions (for example, test at least 3 distinct model families and 5 presentation formats per quarter). Use the ACES‑style controlled setup to measure selection frequency by SKU and attribute.
- Log provenance: for any agent‑mediated decision capture a timestamped payload that includes model vendor and exact version, the full prompt/system message, all external snippets and their sequence order, bundled vs sequential flag, and the “memory” state.
- Set monitoring thresholds: create alerts for changes larger than a practical margin (for example, >10 percentage points change in selection frequency for top SKUs after an upstream model update or content‑format change).
- Force review for high‑value buys: require explicit user confirmation or a veto window for purchases above a value threshold you define (e.g., >$200) until you can demonstrate robust, repeatable agent behavior across models.
- Test seller‑side defenses: simulate adversarial or optimized seller copy with your audit rigs to see how easy it is for rewritten listings, endorsements, or badges to shift your products’ algorithmic share.
- Plan for model drift: treat every upstream model upgrade as a potential behavioral change event and re‑run audits immediately after vendor releases.
Key questions, concise answers
- Can a single review page shift an AI agent’s purchase?
Yes. In the ACES experiments, a Wirecutter review favoring the Fitbit Inspire 3 increased selection probability by about +90 pp for Claude Opus 4.8 and +99 pp for Gemini 3.5 Flash versus a no‑source baseline. - Does adding more recommendation snippets make choices more reliable?
No. Adding sources did not average out choices; a dominant endorsement (Wirecutter in these tests) often remained decisive and overall variability increased when multiple sources were present. - Do short user “memory” lines change agent decisions?
Yes. Short preference snippets like “I love hiking!” shifted several models away from the objectively superior, low‑cost option toward pricier watches (example lifts: +75 pp, +37 pp, +36 pp for specific models reported by the authors). - Are all models equally manipulable?
No. The experiments show heterogeneity. For instance, Gemini 3.5 Flash selected the objectively best product in about 86-92% of runs regardless of memory injections, while others showed much larger swings. - Should businesses optimize content for AI shoppers the same way they do SEO for humans?
Not yet. Model diversity, sensitivity to presentation order and bundling, and opaque upstream inputs mean there’s no single “AI SEO” playbook. Optimization is possible, but it’s brittle and requires continuous, model‑specific testing.
Final thought
Agentic shopping is real and promising, but not yet a plug-and-play replacement for human oversight. The Wharton ACES experiments show that AI buyers interpret endorsements, layout, and tiny preference cues through the particular lens of their architecture and prompts, and that produces systematic, model‑dependent shifts rather than a single uniform notion of “best.” These results come from a simulator using fixed screenshots and curated snippets, so treat them as a warning light about fragility and a call to build robust auditing, provenance, and human‑in‑the‑loop controls before you let agents transact at scale.