Even frontier LLMs from GPT-5 onward lose up to 33% accuracy when you chat too long

TL;DR — Leading LLMs still lose a large chunk of accuracy when tasks are split across many chat turns instead of provided as one prompt: earlier tests showed about a 39% drop; the newest generation reduces that to roughly a 33% loss. Practical fixes exist now—periodic summarization + session resets, retrieval-augmented grounding, and canonical state storage—so teams should instrument and deploy them rather than wait for architecture-level cures.

The problem, plain and simple

“Sharded” inputs—when you split a task across many chat messages instead of sending all instructions at once—cause LLMs to perform worse than when the same content is concatenated into a single prompt. That matters because most human workflows are iterative: users clarify, change requirements, and add caveats over dozens of messages. Researchers led by Philippe Laban tested frontier models (GPT-5 and similar) across six task families and found a consistent multi-turn degradation that remains material even in the newest models.

Leading LLMs still struggle when tasks are distributed across many chat turns rather than given in one prompt.

Quick jargon check: “sharded” means inputs split across messages; “checkpointing” means consolidating conversational state into a canonical summary or stored record so the model consumes that consolidated state as its single source of truth going forward.

What the tests showed

• Task families tested: Python/code, databases, actions (API-like workflows), data-to-text, math, and summarization.
• Multi-turn vs single-shot: Splitting input across turns caused large accuracy drops compared with a single concatenated prompt.
• Magnitude: Older baselines showed ≈39% accuracy degradation from sharding; frontier LLMs (GPT-5 and onward) reduced that to ~33%—an improvement, but still large for production use.
• Task variance: Code/Python tasks were most resilient (≈10–20% loss). Reasoning-heavy tasks like multi-step math and summarization showed the biggest hits.
• Tuning knobs: Lowering temperature (which reduces randomness in generation) did not meaningfully fix multi-turn degradation. Temperature controls output stochasticity and was expected to stabilize answers, but the failure mode appears deeper than sampling noise.

Why businesses should care

This isn’t an academic quirk. If your sales assistant, support bot, or automation agent accumulates state across a 30–message conversation and silently degrades accuracy by a third, the outcomes are real: wrong quotes, incorrect invoices, failed automation runs, or confused customers. The tests used relatively simple simulated users—real users who change constraints mid-conversation will likely make the problem worse.

Two short vignettes

Support bot: A customer clarifies billing terms across 25 messages. The bot later suggests a plan that violates a constraint given early in the chat. The chat feels natural, but the result is costly.

Sales configurator: A salesperson iterates on product options across many turns. When the system generates the final quote, it omits an add-on discussed five messages earlier and overcharges the client.

Immediate mitigations: recipes you can implement today

There are pragmatic, low-lift patterns that reduce multi-turn fragility today. They trade a little UX friction and token cost for much better reliability.

1) Summarize-then-reset (checkpointing)

When a session grows long or a critical variable changes, generate a concise canonical summary of the conversation and start a fresh session that contains the system prompt + the summary + the current user intent. Treat the summary like a committed transaction.

Heuristics for triggering a checkpoint:

• After N turns (e.g., 8–12) or when messages exceed a token threshold (e.g., 4–8k tokens).
• On any edit to an earlier instruction or when a task-critical variable changes (price, scope, credentials).
• When the model repeatedly asks clarifying questions (3+ loops) or when success rate drops in monitoring.

Lightweight pseudocode:

DetectCheckpoint() → if turns > N or tokens > T or critical_change then

  summary = Model.summarize(chat_history, system_instructions)

  storeCanonicalState(summary)

  startNewSession(system_prompt + summary + current_user_message)

Sample system prompt to generate a summary:

“Summarize the customer’s constraints, decisions, and open questions in 4–6 bullet points, emphasizing the final agreed variables and any unresolved items.”

2) Ground on stored canonical state with RAG

Combine summarization with retrieval-augmented generation (RAG): store important facts in a small knowledge store and retrieve them into the prompt rather than relying on chat history alone. This reduces attention dilution from long histories.

3) Use canonical state storage separate from conversational text

Keep structured facts (SKU selections, price caps, customer IDs) in a database and surface them to the model when needed. Treat the chat as the UI and the canonical state as the source of truth.

Monitoring, validation and an A/B test plan

Instrumenting multi-turn failure modes is essential. Useful metrics and checks:

• Task success rate by turn count: bucket sessions by length and compare success.
• Clarification loop count: average number of follow-up questions per session.
• Response consistency: periodically re-run key prompts and compare outputs for stability.
• Token & latency impact: monitor cost and response latency after concatenation or RAG retrieval.

A/B test suggestion:

• Variant A: baseline chat agent.
• Variant B: same agent with automatic summarization checkpoints after 10 turns and RAG grounding for core facts.
• Measure: task completion, user satisfaction (NPS/CSAT), average tokens, and operational cost.

Trade-offs every leader should weigh

Checkpointing improves accuracy but increases token consumption and may add a small latency or UX step (a brief “I’m summarizing and saving your preferences” moment). Summaries can lose nuance; critical flows may still need a human-in-the-loop verification. Large context or persistent memory models will reduce some pain points over time, but those architectural improvements are not yet a universal panacea.

Long-context models, memory APIs, and better dialog-specific training objectives are promising future fixes. Until they are widely available and proven, consolidate state proactively.

How to validate this in your stack

1. Run a small test suite across your most important flows using both concatenated single-shot prompts and natural multi-turn interactions; measure the accuracy gap.
2. Implement automatic summarization and re-run the suite with checkpointing enabled.
3. Compare outcomes, token costs, and UX signals; then roll the fix to high-value flows first (billing, quotes, legal outputs).

Action checklist for product teams and execs

• Instrument chat length vs task success now.
• Implement automatic summarization checkpoint for high-risk flows.
• Store canonical state and surface it via RAG or structured retrieval.
• Run an A/B test comparing baseline vs checkpointed flows.
• Define escalation: human review for any automated summary that changes critical variables.

How to explain this to your CEO / board (2 sentences)

Multi-turn chat interactions can reduce LLM accuracy by roughly a third compared to single-shot prompts, creating real operational risk for billing, sales quotes, and automated decisions. Immediate mitigation: instrument multi-turn success metrics and deploy periodic summarization checkpoints for your highest-value conversational workflows.

Where this came from

Researchers led by Philippe Laban measured multi-turn degradation across frontier LLMs and task families; reporting and context were summarized by Matthias Bastian for The Decoder and highlighted publicly on social platforms. The practical pattern—summarize, concatenate, and proceed—aligns with prompt engineering, RAG, and canonical-state design that product teams already use in AI automation.

Final thought

Conversation feels natural to humans, but LLMs still prefer well-committed inputs. If accuracy matters, design conversations like transactions: commit state frequently, validate often, and treat long chats as fragile processes that need checkpoints.

Sources & further reading: Philippe Laban’s research highlights and related coverage on The Decoder; general resources on retrieval-augmented generation (RAG) and LLM session management from vendor docs and community primers.