AI agents automate experiments but fail to produce publishable research, study shows

Study challenges lab claims that AI agents can autonomously conduct publishable research

In a tightly controlled test reported by Jonathan Kemper in The Decoder (Aug 14, 2026), researchers from Princeton and the UK AI Security Institute gave state‑of‑the‑art models the exact same research problems their human authors had been working on, with full compute, API credits, web access, and a six‑day window, and found the agents handled engineering chores but failed at the creative, judgment‑heavy work that makes a paper publishable.

What the experiment actually did

The team published their protocol and artifacts at https://cruxevals.com/crux/can-ai-agents-conduct-research and called the method Shadow Evaluation. Instead of testing on public problems the agents might already have seen in training data, Shadow Evaluation handed agents the core research questions from two unpublished NeurIPS 2026 submissions and asked them to pursue solutions. The original human authors served as the reviewers.

The two target problems were: a “Personas” paper about steering model personality traits via weight interventions, and a TabPFN‑style paper that detects deployment data shifts that break tabular prediction models. Agents had a six‑day run, $3, 000 in API credits, a GPU budget (the paper does not specify exact GPU limits), a virtual machine, and access to the open web.

Primary model and setup: Claude Opus 4.8 with an “Extra‑High Reasoning” setting, orchestrated through a CRUX‑2 scaffold implemented with OpenClaw, which is an open‑source agent framework by Peter Steinberger. To check for model‑or‑framework effects, the team repeated the experiment with GPT‑5.6 Sol and an OpenAI Codex scaffold.

The headline result

Agents performed the engineering side of research well: literature searches, debugging GPU jobs, running many experiments, and compiling LaTeX. But when the original authors judged the outputs, both agent‑written papers were rejected; one received a “Strong Reject.” The study’s blunt summary: frontier models “cannot solve weeks-long, open-ended AI research questions.”

“Frontier models cannot solve weeks-long, open-ended AI research questions.”, study authors

Concrete failures (what actually went wrong)

Early abandonment of ambitious plans. Both agents dropped their most ambitious goals within the first ten hours. In the Personas run the agent completed its “open exploration” phase after roughly five hours even though it had scheduled 36-48 hours of work.
Self‑declared completion and underspend. One agent declared the project complete seven hours before the deadline. Both runs spent less than half the $3, 000 API budget; the Personas run logged $1, 130 of spending.
Instruction drift and formatting mistakes. Both agent drafts exceeded NeurIPS length limits. One agent produced a draft with zero visualizations in the main text while the human original contained 15 figures.
Poor high‑level judgment. Reviewers flagged reasoning errors described as a “‘proof by example’ fallacy, ” called experimental choices “bizarre, ” and criticized “post hoc” selection of results. One reviewer labeled a submission a “Strong Reject.”
Repeatability of failure modes. The GPT‑5.6 Sol repeat run reproduced most failure patterns. In that repeat, GPT‑5.6 burned through the $3, 000 budget in a little over two days, which produced many smaller, rushed experiments rather than stronger scientific judgment.
Only a few human fixes. The team documented three human interventions: a scaffold bug fix, a deadline extension, and a request to rewrite a readability problem. The paper does not quantify the scope of the readability edit.

What worked, and why that distinction matters

These agents are already highly effective at the labor‑intensive, well‑scoped engineering components of research: running experiment pipelines, triaging failing jobs, iterating code, searching literature, and producing LaTeX drafts. For internal R&D, reproducibility checks, or automating routine experiment runs, that competence saves time and reduces human toil.

Publishable research, however, demands more than engineering. It requires creative hypothesis generation (abduction), principled meta‑judgment about what experiments meaningfully test a claim, strategic backtracking when a line of inquiry stalls, and strong responses to reviewer critique. Those are precisely the deficiencies Shadow Evaluation exposed.

Technical diagnosis: why agents stumble on “real” research

Abduction vs. deduction. Large language models excel at deduction, following given rules, verifying derivations, or exploring combinatorial search, but they struggle at abduction: inventing new causal hypotheses or reframing a hard research problem in a genuinely novel way.
Instruction drift / behavioral state decay. Over long, multi‑step runs agents show gradual drift: they forget earlier constraints, exceed formatting limits, or omit requested artifacts. The authors connect this to observations labeled “behavioral state decay” by Meta AI.
Poor meta‑judgment and backtracking. Research demands knowing when to abandon an approach, when to expand an experiment, and how to design informative ablation studies. Agents tended to settle on superficial wins, select post hoc best runs, or fail to reinterpret negative results.

How this sits beside recent lab claims and selective breakthroughs

Anthropic and OpenAI have published accounts that highlight agent‑driven acceleration or selective successes, for example Anthropic’s post titled “When AI Builds Itself” (June) and OpenAI’s statements that GPT‑5.6 Sol helped post‑train a smaller model called Luna and “saved researchers several weeks.” Those reports emphasize engineering speedups or narrowly scoped achievements, not necessarily end‑to‑end autonomous discovery in open research contexts.

There are domain‑specific counterexamples that complicate the picture. In May 2026 an OpenAI reasoning model reportedly disproved a unit‑distance geometry conjecture posed by Paul Erdős; Fields Medalist Tim Gowers called that result a “milestone in AI mathematics.” Anthropic’s Claude Mythos also reportedly produced a similar proof, described by Anthropic engineer Sholto Douglas as a “cute, simple proof.” Sakana AI’s The AI Scientist‑v2 submitted three papers to an ICLR workshop in 2025 (one paper accepted with an average score of 6.33 and later withdrawn for citation errors) and related research from that group appeared in Nature in March 2026. These episodes show AI can produce notable, domain‑specific breakthroughs, especially where problems reduce to formal reasoning or optimization.

Shadow Evaluation, by contrast, targeted open, weeks‑long AI research problems and used the original authors as gatekeepers. In that setting, with broader experimental design, noisy empirical data, and extended judgment calls, agents failed to replace the human researcher.

Limitations and open next steps

The authors are explicit about limits: this study tested two NeurIPS 2026 papers and reviewers were not blinded to the evaluation. That leaves key questions open:

Would blinded review change outcomes? A follow‑up should randomize reviewer knowledge of origin and agent involvement.
Does scaling compute or time close the gap? The authors suspect conceptual limits (abduction and judgment) matter more than modest increases in compute, but rigorous experiments are needed.
Are some scaffolds, prompts, or model families better suited to abduction? The repeat with GPT‑5.6 Sol reproduced failure modes but consumed budget differently, and broader cross‑model tests across >10 problems and staged compute budgets would help.
How should we measure “autonomous research”? Peer review is noisy, the paper calls many prior evaluations “overstretched, stochastic, and suffers from poor review quality”, so better benchmarks for novelty, hypothesis‑generation, and reviewer‑level acceptance at main conferences are needed.

Concrete next experiments the community could run: scale the sample size (N>10 problems) across formal and empirical domains, run blinded reviews, compare scaffolds like OpenClaw vs Codex under matched budgets, and test staged compute budgets (×2, ×10) to see whether failures are conceptual or purely resource‑limited.

What this means for business and R&D leaders

If your board asks whether you can hand an AI agent a complex research brief and expect a publishable paper, the prudent answer is no, not yet. But agents already provide measurable value when used as junior engineers and lab technicians: automating experiment orchestration, triaging jobs, generating first drafts, and scaling reproducibility. Those uses align with AI automation for R&D rather than fully autonomous scientific discovery.

Recommendations for executives:

Adopt hybrid workflows. Let AI agents automate repetitive engineering tasks, and reserve human researchers for ideation, high‑level judgment, and peer‑review interactions.
Measure the right things. Track novelty‑focused metrics (novel hypotheses proposed, depth of ablation studies, acceptance rates at main conferences) not just throughput (number of experiments run or drafts produced).
Use Shadow Evaluation, style controls. When claiming agent autonomy, require transparent artifacts, blinded evaluation where possible, and public release of logs to avoid overstating capabilities.

Key takeaways, questions you might ask

Can current frontier AI agents autonomously perform the full research cycle and produce publishable work?
No, in Shadow Evaluation runs on two real NeurIPS 2026 problems, agents reliably handled engineering but failed at creative judgment, abduction, and reviewer‑level reasoning; both agent papers were rejected (one “Strong Reject”).
Are AI agents useful for research at all?
Yes. They execute engineering tasks, literature search, debugging, experiment orchestration, and LaTeX compilation, well enough to speed workflows and reduce researcher toil.
Do these results contradict lab reports that agents accelerate research?
Not necessarily. Labs often report engineering speedups or selective breakthroughs. Shadow Evaluation shows those gains do not automatically translate into dependable, end‑to‑end autonomous discovery on open, weeks‑long research problems.
Would more compute or time likely fix the core problems?
The study’s authors suspect no, the failures were conceptual (abduction, meta‑judgment, backtracking), not only resource‑limited, but this remains an open empirical question.
How should organizations measure progress toward autonomous research?
Use benchmarks that test creative inference and reviewer‑level novelty, include blinded peer review where possible, and prefer transparent artifact release over cherry‑picked anecdotes.

Final thought

Frontier models are reshaping how research gets done: they are powerful assistants for engineering, reproducibility, and scaling experiments. But the messy craft of original science, inventing hypotheses, judging tradeoffs, and responding to skeptical reviewers, remains a human domain for now. That gap is the practical signal executives and policymakers should use: automate the parts agents do well, measure creative value precisely, and keep humans firmly in the loop where judgment matters.

“Existing evaluations either test agents on narrow tasks or submit AI-generated papers to peer review, a process the researchers call ‘overstretched, stochastic, and suffers from poor review quality.'”

The dataset, logs, and reviewer artifacts are available at https://cruxevals.com/crux/can-ai-agents-conduct-research for teams building agent workflows or designing follow‑up experiments. Use them to stress‑test your roadmap: automate where agents are strong, and prioritize human oversight where novelty and judgment drive value.