Can LLMs engineer their own harness? HarnessDev finds only 34 of 64 changes moved feedback and held‑out scores together
One creator LLM added 17, 111 net lines of code to its harnesses. Another added just 1, 006 lines and still led a terminal benchmark. That gap (more code does not equal more robustness) is the clearest image from HarnessDev, a fresh evaluation that asks models to write the code that runs them.
“An agent harness is the code around a model: execution loop, tools, context, state, recovery, and verification.”, HarnessDev (ByteDance Seed et al., Sep 2026)
What HarnessDev changes about evaluation
Most benchmarks judge a model’s answer inside a fixed harness. HarnessDev flips that. It grades the runnable harness the model builds. The harness is the surrounding code that operates the model, looping, calling tools, managing context and memory, recovering from errors, and verifying results.
The workflow has two phases. Creation is a single‑shot build from a deliberately weak seed (basic file/search/process primitives plus result and trajectory writers) and 1-3 development cases. The harness is frozen before hidden tasks. Evolution lets the same creator iteratively revise that frozen harness using execution feedback from fixed evaluation suites.
Key evaluation terms used below:
- Executor tokens, tokens consumed by the model that acts as the executor at runtime (used to measure runtime cost; creator tokens, the cost to produce the harness, are excluded from efficiency scoring).
- Medal rate, the benchmark success metric reported by the paper’s suites.
- Pair‑score point, the numeric difference between two adjacent harness versions measured on the held‑out tasks (used to quantify commit impact).
- Self‑Eval, running a harness with its creator as executor/runtime.
- Unified‑Eval, re‑running all harnesses under a single fixed executor (Gemini 3.1 Pro) for cross‑model comparison.
The paper: HarnessDev (ByteDance Seed et al.), arXiv:2609.01437; project page at self-developing-agents.github.io.
Who was tested, at a glance
- Creators: Opus 4.8, GPT‑5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, Seed 2.0 Pro.
- Execution environments: during creation and evolution each creator ran in its native executor (most inside Claude Code 2.1.177, GPT‑5.5 used Codex 0.144.3). For unified cross‑model comparison all harnesses were later re‑run under Gemini 3.1 Pro.
- Benchmarks: five suites totaling 2, 207 instances, SWE‑bench Pro (public split 731; 630 held‑out instances used later), Terminal‑Bench 2.1 (89), MLE‑bench (75), EQ‑Bench3 (46), BrowseComp (1, 266).
Bottom line: strengths, weaknesses, and brittle plumbing
Bottom line: models can write runnable harnesses that execute and achieve nonzero task success, but results vary a lot by domain, executor, and how much of the generated scaffolding actually runs.
Strengths
- Writing and ML‑experimentation tasks: creators matched or beat human references. Opus 4.8 scored 84.6 on EQ‑Bench3 (versus reference 83.7). On MLE‑bench Opus 4.8 reached 32.9 and Gemini 32.4, both above the reference 24.0.
- Creation from a weak seed worked. The unmodified seed scores 0 everywhere. Creators produced harnesses that executed and gained nonzero task success on at least one benchmark.
Weaknesses
- Code and search/browser tasks lagged. The best BrowseComp result was 52.6 (GPT‑5.5) versus a reference 92.2. Terminal‑Bench and SWE‑Pro often trailed human references. (Note: some SWE‑Pro, Terminal‑Bench, and BrowseComp references are external results from OpenAI’s GPT‑5.6 report, not re‑runs by HarnessDev.)
- Executor sensitivity. Behavior depended strongly on runtime. For example, Opus 4.8’s SWE‑Pro self‑eval fell from 69.3 to 33.0 under the unified Gemini executor after a swap, because one harness had hard‑coded a 120‑step limit tuned to its original runtime. Terminal‑Bench leaderboards also show large executor differences, for example 35.2% versus 49.6% success inside different CLIs for the same model weights.
- Cost variance. Executor token use varied sharply. On MLE‑bench the paper reports roughly 7× difference in executor tokens for similar medal rates (29.3M → 208.4M), showing dramatically different running costs for comparable performance.
Dead code, inert memory, and shallow wiring
Many generated harnesses include components that never execute, especially state and memory scaffolding. That matters. Scaffolding that never fires can create a false sense of capability. Long term memory can exist on paper but not in practice.
- Of 108 generated code component instances, 72 (67%) triggered in real runs while 18 (17%) never fired. All 18 of the unused components were state and memory related.
- Eleven of 18 harnesses defined a State class, yet no checkpoint events appeared across 26, 679 recorded trajectories.
- 124 of 587 writing features (21%) were dead code. During evolution 169 new functions and classes were added and 25 (≈15%) had no caller.
Those are not cosmetic issues. Dead or unused code increases maintenance surface, hides bugs, and makes portability harder when a harness depends on implicit runtime behavior it never validated.
Evolution helps, but gains are small, noisy, and often non‑transferable
Evolution, iterative revision using execution feedback, produced measurable improvements, but the signal was weak and noisy.
- Nine lineages produced 73 official versions and 64 adjacent switches. All five creators evaluated under their self‑runtime improved on held‑out tasks, with gains from +1.43 to +4.44 points (mean +3.11).
- Under the fixed Gemini executor only Opus improved, while GPT‑5.5 regressed by 10.32 points after the swap.
- Of 64 adjacent switches, only 34 (53.1%) moved immediate feedback and held‑out scores in the same direction. Eight switches regressed on both metrics, 16 regressed on one, 27 were inside the noise band, and just 2 showed clear positive evidence.
- Measurement noise matters. A single commit can vary by about ±4.75 pair‑score points, so many changes sit inside that uncertainty window.
- Only 2 of 9 declared final versions were optimal on held‑out data. Dedicated failure‑diagnosis tooling was rarely used; the trajectory interface was called only twice.
There were effective surgical fixes. For example, Opus 4.8 discovered it reported success in 99 of 100 runs while only 48 actually passed. Adding a completion gate fixed detection. But such clear, generalizable fixes were rare.
Practical implications for teams shipping agents
If you’re building production agents, treat the harness as first class. The model is one ingredient. The harness frequently determines whether the system succeeds in the real world.
Practical steps teams can apply now:
- Test early in the target runtime. Run development harnesses in the same executor your production stack will use. Hard‑coded step limits, tokenization quirks, and tool latency assumptions break portability.
- Instrument trajectories and state usage. Require telemetry that shows whether memory and state classes are exercised. Treat generated memory code as a hypothesis and validate it with checkpoints and counters.
- Track both capability and efficiency. Measure task success alongside executor tokens. Two harnesses with similar pass rates can have wildly different run costs, so plan for ops accordingly.
- Require held‑out validation before promotion. Evolutionary improvements on immediate feedback often fail to generalize. Promote only changes that improve held‑out performance outside the noise band (±4.75 pair‑score points in these experiments).
- Audit for dead and inert code. Automatically detect functions and classes with no callers or components that never trigger in real runs. Prune or fix them before deployment.
- Use explicit completion and verification gates. Don’t rely on implicit signals from the executor. Add checks that detect partial or premature completions.
Limitations and what to watch
HarnessDev is an important step, not the final word. Results reflect six creator models and the executor setups studied. Behavior may differ with other models, richer trajectory interfaces, different budgets, or real external tool APIs. Measurement noise, executor dependence, and varying development environments mean these findings should be read as experimental evidence about harness engineering under the paper’s protocol, not universal laws.
Concise takeaways, quick Q&A
- Can LLMs build runnable agent harnesses from a weak seed?
Yes. All creators in the study produced harnesses that executed and achieved nonzero task success on at least one benchmark, but “runnable” here means the harness runs and completes tasks, not that it matches human references across all domains.
- Do self‑built harnesses match human‑engineered references?
Mixed. Creators matched or exceeded references on writing and ML‑experiment tasks (EQ‑Bench3, MLE‑bench) but underperformed on code‑oriented and search/browser tasks (SWE‑Pro, Terminal‑Bench, BrowseComp).
- Does iterative evolution reliably generalize to held‑out tasks?
Not reliably. Of 64 adjacent changes, only 34 (53.1%) shifted both immediate feedback and held‑out scores in the same direction; many changes remained within the ±4.75 pair‑score‑point noise band.
- How sensitive are harnesses to the executor/runtime?
Very. Swapping executors reshuffled rankings and sometimes halved scores, for example Opus 4.8 SWE‑Pro 69.3 → 33.0. Expect runtime assumptions to break portability.
- Are generated state and memory components actually used?
Often not. 72 of 108 generated code components (67%) triggered in runs while 18 (17%) never fired; all unused components were state and memory related, and no checkpoints were observed across 26, 679 trajectories for harnesses that defined State classes.
HarnessDev reframes a practical truth: the surrounding code matters as much as the model. LLMs can produce useful harnesses, but portability, measurement noise, and inert scaffolding are real engineering risks. Bigger models alone won’t fix brittle agent systems, better harness engineering, clearer diagnostics, and stricter held‑out validation will.
Source: HarnessDev, ByteDance Seed et al., Sep 2026 (arXiv:2609.01437); project page at self-developing-agents.github.io.