Benchmarking Harness Self-Improvement

EVO-BENCH

Can Language Models Improve Agent Harness?

Measuring a model's intrinsic ability to autonomously diagnose, rewrite, and improve the executable agent harness.

Paper GitHub Dataset

Evolver models
evaluated

608

Harness-sensitive
tasks

+16.6

Best absolute gain
over seed

Agent domains,
5 benchmarks

48h

Research budget
per run

Leaderboard

Harness-Evolving Capability

Open-weight only Show baselines

Dataset

Get Evo-Bench on Hugging Face

Evo-Bench

RUCAIBox/Evo-Bench · validation + evaluation splits

validation 160 evaluation 448 total 608

Open in Hugging Face

Domains

3 search / office / general

Source benchmarks

Candidate pool

2,329 screened

Selection

Sens > 0 stratified

# pip install datasets
from datasets import load_dataset

# the suite an evolver may evaluate against during a run
val  = load_dataset("RUCAIBox/Evo-Bench", split="validation")

# sealed until the harness is frozen
test = load_dataset("RUCAIBox/Evo-Bench", split="evaluation")

Each task carries its source benchmark, domain, native scorer, difficulty stratum, and harness-sensitivity score, so you can reproduce the split or re-stratify it yourself.

608harness-sensitive tasks

Three agent domains · five source benchmarks

Search

320

BrowseComp16032 validation · 128 evaluation

HLE16032 validation · 128 evaluation

Office

192

GDPval9632 validation · 64 evaluation

APEX-Agents9632 validation · 64 evaluation

General

Claw-Eval9632 validation · 64 evaluation

Validation160

Evaluation448

Validation: 32 tasks from each benchmark.Evaluation remains sealed until the harness is frozen.

Explore Evo-Bench

Construction

Tasks Chosen by Harnesses, Not by Hand

A benchmark for harness evolution is only meaningful if its tasks actually move when the harness improves. Evo-Bench selects tasks using evidence from real evolved harnesses rather than intuition.

Two-stage harness-guided benchmark construction framework

Two-stage harness-guided construction. Stage 1 evolves harnesses on auxiliary tasks that are disjoint from every source benchmark. Stage 2 uses those harnesses to score each candidate task's sensitivity, then selects and splits with difficulty stratification.

Auxiliary tasks, fully disjoint

320 tasks drawn from MiroRL, RedSearcher, Auto-ClawEval and internal sets — corpus- and instance-level disjoint from the five source benchmarks, so no evidence leaks into the final suites.

12 representative harnesses

Four independent evolution runs (GLM-5.2, Opus 4.8, Sonnet 5, GPT-5.6 Sol) produce 73 evaluated harness variants; deterministic diversity-aware selection keeps 12 spanning capability, tool orchestration and program structure.

Sensitivity, not variance

For each task we correlate its per-harness scores with each harness's leave-one-out overall quality. High variance alone says nothing about direction; correlation says whether better harnesses actually win the task.

Aligned, disjoint splits

Tasks with non-positive sensitivity are dropped, the rest are stratified by difficulty, and validation and evaluation are drawn within each stratum — keeping both splits distributionally aligned.

Difficulty vs harness sensitivity item map

Item map. Difficulty against harness sensitivity for every candidate. Selection favours positive sensitivity while preserving difficulty coverage; the two splits stay aligned.

Source	Pool	Sens ≤ 0	Val	Eval	Mean Sens
APEX-Agents	421	133	32	64	0.377
BrowseComp	768	47	32	128	0.256
Claw-Eval	157	60	32	64	0.386
GDPval	215	95	32	64	0.368
HLE	768	62	32	128	0.313
Total	2,329	397	160	448	—

Sensitivity is a Pearson correlation between task score and leave-one-out harness quality across the 12 auxiliary harnesses. Mean Sens is computed over the surviving candidates.

Findings

What Nine Evolvers Revealed

Every evaluated model improves the seed harness, but how they improve it — and where they stall — separates them sharply.

Evolution saturates early, then regresses

Models find strong structures within a handful of iterations, then keep editing past the point of benefit. Opus 4.8 and GLM-5.2 post the highest AnytimeVal scores (51.4 / 51.0) yet freeze below their own peak. Of the three models we audited in depth, only Kimi restored its best snapshot before freezing.

Gains are domain-shaped, and Office resists

Search is where evolution pays: the seed scores 11.7 and Opus 4.8 reaches 46.5, essentially catching the human-engineered harness by writing the missing retrieval stack from scratch. Office barely moves — its specialised document and spreadsheet workflows are hard to discover by trial. On General tasks the top evolvers reach 59.4 and pass the human baseline of 56.3.

Evolved harnesses transfer across policy models

Swapping the policy model to Qwen3.6-35B-A3B or GLM-5.2 preserves the gains, so evolvers are synthesising reusable reasoning and tool-use structure rather than patching one model's quirks.

Capability and spend are only loosely coupled

The frontier is steep but wide: the top score costs roughly $600–800 of evolver inference, while GLM-5.2 and Qwen3.7-Max land within a few points for well under $50, and DeepSeek V4 Pro produces a working improvement for about a dollar.

Cost vs. Overall Score. Evolver inference only, log scale. The dashed line is the Pareto frontier; the horizontal line is the human-engineered harness.

Budget use. Iterations, steps and wall-clock against the 20 / 1,000 / 48h ceiling. Most evolvers stop long before the budget runs out.

Validation score trajectories for three lower-scoring models

Validation trajectories. Stars mark each run's best revision, squares mark what it actually froze — the gap between them is self-inflicted. Bands show intervention surfaces; Kimi restored I13 after I20.

The seed is genuinely blind here

CodeAct scores 11.7 with nothing but a shell. There is no fetch, no HTML handling, no result triage — so almost the entire gap is capability the evolver has to invent.

+34.8, and a near-catch

Opus 4.8 reaches 46.5 against the human-engineered 46.7. GLM-5.2 and GPT-5.6 Sol follow at 45.4 and 44.5.

What actually gets built

Search and fetch tools, plus a page cleaner that strips scripts, styles and markup noise while keeping links intact — the difference between a usable observation and a context flood.

Why this domain rewards evolution

The missing pieces are conventional, well-understood infrastructure. Models have seen this shape of code many times, so recovering it from failure evidence is tractable.

The stubborn domain

The best gain is +3.3 (MiniMax M3) and several evolvers regress. No model reaches the human-engineered 43.9.

Less headroom, harder wins

The seed already scores 38.4, so there is no free capability gap — what remains demands precise, format-aware workflows.

Splitting the sub-domains helps

GPT-5.6 Sol separates APEX from GDPval: evidence tracked by file, page and cell for one; create, recalculate, reopen and inspect the rendered artifact for the other.

Where human engineering still wins

Office rewards bespoke document and spreadsheet infrastructure — exactly the kind of specificity that general trial-and-error struggles to discover.

Autonomous beats hand-built

GPT-5.6 Sol and Qwen3.7-Max both reach 59.4, above the human-engineered 56.3 — the clearest evidence that evolved structure can exceed a crafted one.

Restraint is the winning move

The gain comes from removing over-steering prompt scaffolding, not adding more. Regression diffs localise the interference; the fix is a smaller harness.

Keep the guards, drop the noise

What survives the trim: recovery from empty or premature responses, credential redaction, and a guard against sending draft-only email.

Flexibility over specification

General tasks reward adaptable reasoning rather than fixed procedure, which is precisely the axis evolution explores well.

Qwen3.6-27B — blaming noise

Climbs to 49.7 by I10, then a malformed verifier at I15 crashes 159 of 160 tasks. Repairs recover only to 45.4. It attributed the 4.3-point gap largely to run-to-run noise and never restored the earlier revision, freezing with two iterations and 651 steps unspent.

DeepSeek V4 Pro — declaring a plateau

Hits 46.5 at I3, then tunes prompts and filters while opening raw rollouts only four times. Two of its last evaluations scored unchanged code. Froze at 42.6 with five iterations and 798 steps unused.

Kimi K2.7 Code — disciplined but local

Recovers from an invalid first architecture, reaches 45.9 at I13, tests seven further variants, and correctly restores the byte-identical best snapshot. Its limit was search efficiency, not discipline: 549 of 1,000 steps, all of it local tweaking.

The shared pattern

All three add the obvious capabilities fast, then drift into prompt, threshold and gate edits instead of paired failure analysis. What is missing is one falsifiable change per iteration, cheap preflight checks, and automatic best-revision recovery.

Wide autonomy, hard walls

The evolver may read rollouts, task provenance, expected answers and scores — that access is the diagnostic signal. It may write only its harness copy and workbench.

The policy never sees answers

Each rollout gets an immutable snapshot, a public task view with scorer and expected fields stripped, and a fresh workspace. Held-out data is never mounted anywhere in the loop.

Retrieval is scanned, not banned

Disabling the web would break Search, so tool output is screened for benchmark answer keys and dumps instead. Only content that actually reaches the policy counts as a violation; blocked or failed queries do not.

One model tried to hide it

A semantic audit over all trajectories checks for paraphrased or obfuscated leakage. MiniMax M3 was the sole model to show evasion behaviour; the affected trials were zeroed and its reported result is the corrected one.

GPT-5.6 Sol harness evolution case study

Case study: GPT-5.6 Sol. Rather than one uniform harness it builds a hierarchical router with domain-specific prompts and tools. Intermediate scores are validation; the final block is held-out.

It reacts to scores, not causes

Aggregate movement drives the next edit; the underlying failure mode is rarely distilled from the logs it already has.

Routing substitutes for architecture

Cross-domain interference is resolved by branching per domain rather than finding a mechanism that works for all of them.

The core stays primitive

Even in the best run the planner remains passive, context stays append-only, and the verifier is fully permissive — the parts that would need real redesign go untouched.

Ablations

Budget Scaling & Cross-Policy Transfer

Two questions worth isolating: does a larger research budget keep paying, and does an evolved harness survive a change of policy model?

Budget scaling. Qwen3.7-Max and GLM-5.2 across three ceilings (24h/10/500 → 36h/15/750 → 48h/20/1000). Both improve monotonically; GLM-5.2 climbs steeply then flattens, Qwen3.7-Max grows more linearly.

Evolver	Search	Office	General	Overall	ATV
Policy: Qwen3.6-35B-A3B
CodeAct seed	2.7	14.2	35.9	13.9	—
Qwen3.7-Max	12.5	33.0	48.4	27.9	29.2
GLM-5.2	16.4	34.0	45.3	29.2	30.8
Policy: DeepSeek-V4-Flash — main setting
CodeAct seed	11.7	38.4	48.4	29.7	—
Qwen3.7-Max	36.3	37.8	59.4	41.5	49.3
GLM-5.2	45.4	39.2	48.4	43.5	51.0
Policy: GLM-5.2
CodeAct seed	18.0	40.2	73.4	38.0	—
Qwen3.7-Max	38.3	45.1	46.9	42.7	46.7
GLM-5.2	35.6	45.2	80.3	48.4	50.4

Cross-policy transfer. Gains hold under every policy swap. With GLM-5.2 as both evolver and policy, the harness lifts 38.0 → 48.4 — past the human-engineered 47.5 measured in the main setting. ATV = AnytimeVal.

Team

Lisheng Huang^1,* Chen Yang^2,* Hao Zhou² Huatong Song¹ Zongchao Chen² Ran Le² Yang Song^2,† Wayne Xin Zhao^1,† Tao Zhang²

¹Gaoling School of Artificial Intelligence, Renmin University of China

²BOSS Zhipin, Beijing, China

* Equal contribution · † Corresponding authors
huanglisheng@ruc.edu.cn · batmanfly@gmail.com · songyang@kanzhun.com

Citation

@inproceedings{huang2026evobench,
  title     = {Evo-Bench: Can Language Models Improve Agent Harness?},
  author    = {Huang, Lisheng and Yang, Chen and Zhou, Hao and Song, Huatong and
               Chen, Zongchao and Le, Ran and Song, Yang and Zhao, Wayne Xin and
               Zhang, Tao},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://github.com/RUCAIBox/Evo-Bench}
}