The harness (still) matters

Evidence review

The harness (still) matters

Public studies show that the agent harness changes performance, cost, token use and runtime. They do not identify one harness that wins across every model and task.

Evidence captured on 10 September 2026

Executive summary

Current observations

  • Harness and runtime choices can materially change results. Claw-SWE-Bench found 12.5- to 27.4-point spreads; Harvey's intervention ranged from 0.9 points worse to 23 points better.
  • No harness wins across all models and tasks. Native CLIs led 4 of 6 current Terminal-Bench pairs, while alternative harnesses led elsewhere.
  • Efficiency differences are often clearer than quality differences. Databricks, GitHub and several smaller studies found large cost, context or token gaps beside modest quality changes.
  • The count is not 27 independent replications. Some studies reuse benchmark families or live leaderboards, and many rely on a single attempt per task.
  • We separate matched harness evidence from same-label system comparisons. Provider routing and model snapshots can still differ even when the published model label matches.
  • Teams should test harnesses on their own work. Compare at least 2 harnesses with the same model, effort and budget. Measure quality, cost and time.

Summary overview

What each benchmark tells us

1

Operational and broad evidence

  1. 01Databricks coding-agent benchmark

    Across 6 matched comparisons, Claude Code or Codex cost 1.20 to 2.08 times as much as Pi. Quality stayed within 7 points.

  2. 02GitHub Copilot harness evaluation

    Across 20 matched pairs, Copilot CLI moved task resolution by -8.0 to +7.1 points and used fewer median tokens in 18.

  3. 03Terminal-Bench 2.1

    The native CLI led 4 of 6 current matched pairs; 1 was effectively tied and Terminus 2 led Gemini 3 Pro.

  4. 04Coding Agent Index

    OpenCode led with Opus 4.7; Codex led Cursor with GPT-5.5. In both comparisons, the higher-scoring harness cost more.

  5. 05Agents' Last Exam

    GPT-5.5 full-pass rates spanned 19.1% to 24.0%, while published total cost spanned $180 to $570.

  6. 06DrugDiscoveryBench

    mini-SWE-agent scored highest with GPT-5.5. Codex placed in the middle.

  7. 07Claw-SWE-Bench

    The harness spread was 12.5 points with GLM-5.1 and 27.4 points with Qwen3.6-Flash.

  8. 08Harness-Bench paper

    Across the same 8-model pool, average combined score ranged from 52.4 for OpenClaw to 76.2 for NanoBot.

  9. 09WANDR system comparison and delivery ablation

    A contextual production-system comparison showed a large quality-cost trade-off; within-system ablations favoured file or share delivery over output tokens.

  10. 10Warden benchmark

    Pi found a similar or larger share of known vulnerabilities at about one-quarter to one-fifth of Claude SDK's cost.

  11. 11Agent Security League

    Cursor led on Opus 4.7 and 4.8. Claude Code led on Sonnet 5.

  12. 12DeepSeek V4.1 Flash scaffold evaluation

    With DeepSeek V4.1 Flash fixed, the scaffold spread was 8.7 points on DeepSWE and 6.5 points on Terminal-Bench.

  13. 13Browser Use Pi concurrent revision comparison

    A bundled runtime revision improved BU Bench V2 by 7.53 points, but tied on Hard and failed the study's 2-benchmark confirmation rule.

2

Focused domain evidence

  1. 14Harvey LAB harness ablation

    The same harness intervention ranged from 0.9 points worse to 23 points better, depending on the model.

  2. 15Harness optimization on LAB

    With DeepSeek V4 Pro fixed, pooled criterion pass rate ranged from 3.5% to 80.1% across 5 harnesses.

  3. 16LatchBio behavioural-prior study

    Point-estimate ordering changed by task horizon, although the harness intervals overlapped; a prompt-only ablation moved in opposite directions.

  4. 17CSI scaffold comparison

    CSI::Claude and CSI::Codex both solved 15 of 33 tasks, but Codex cost one-third as much and finished faster.

  5. 18Vulnerability-Agent-Bench

    Claude Code and Codex beat OpenCode on all 3 published matched-model pairs.

  6. 19WildClawBench

    The best harness changed by model. Switching harness moved scores by double digits.

  7. 20CORE-Bench Hard

    Claude Code more than doubled CORE-Agent's verified success with Opus 4.5 while costing less.

  8. 21Prime Agent long-context comparison

    Prime Agent had the higher point estimate in 20 of 27 high-effort pairs, without published uncertainty.

3

Small and diagnostic evidence

  1. 22The Scaffold Effect

    Goose used about 41 times fewer reported tokens than OpenCode for similar pass rates on 2 fixed models.

  2. 23HAL SWE-bench Verified Mini

    SWE-Agent usually solved more tasks than HAL Generalist, but often at several times the reported cost.

  3. 24HarnessBench

    Several configurations finished within one task of the leader. Runtime differed more clearly.

  4. 25A-CODE-CLI

    OpenCode had a small backend lead over Claude Code and cost 44% less with Sonnet 4.6.

  5. 26OpenBench M4.5

    All 4 harnesses passed every task. Pi and Cursor used less time and fewer tokens.

  6. 27Harness Tax

    Fixed input overhead ranged from about 2,600 tokens for Pi to 27,000 for Claude Code.

01Operational and broad evidence

Databricks ran 6 matched harness comparisons: Claude Code or Codex cost 1.20 to 2.08 times as much as Pi, with quality within 7 points.

Published results

Swipe charts horizontally to see every label and value.

Databricks matched harness comparisons Six matched-model comparisons show Claude Code or Codex costing 1.20 to 2.08 times as much as Pi, with task completion within 7 percentage points. Four bars show lower median context use for Pi. Relative cost and task completion 0.00×0.25×0.50×0.75×1.00× Opus 4.8 · high Claude Code ÷ Pi 2.08× Pi · 85% Claude Code · 87% Opus 4.8 · xhigh Claude Code ÷ Pi 1.46× Pi · 90% Claude Code · 88% Opus 4.8 · max Claude Code ÷ Pi 1.20× Pi · 82% Claude Code · 89% GPT-5.5 · medium Codex ÷ Pi 1.54× Pi · 83% Codex · 80% GPT-5.5 · high Codex ÷ Pi 1.22× Pi · 81% Codex · 83% GPT-5.5 · xhigh Codex ÷ Pi 1.44× Pi · 78% Codex · 80% Relative cost · Claude Code or Codex = 1.0 Median context re-fed per task 00.35m0.70m1.05m1.40m Opus 4.8 · Pi 236k Opus 4.8 · Claude Code 742k GPT-5.5 · Pi 665k GPT-5.5 · Codex 1235k Context tokens

The source publishes exact cost ratios and scores, but not exact dollar values for each matched point. Costs are indexed to Claude Code for Opus and Codex for GPT.

Back to benchmark map

02Operational and broad evidence

GitHub compared Copilot CLI with Claude Code or Codex across 5 suites: quality was mixed, while Copilot used fewer median tokens in 18 of 20 pairs.

Published results

Swipe charts horizontally to see every label and value.

GitHub Copilot task-resolution and token changesFour panels plot Copilot CLI's task-resolution difference against its median token difference from Claude Code or Codex. Most points use fewer tokens and quality differences range from minus 8 to plus 7.1 percentage points.Claude Sonnet 4.6-70%-50%-30%-10%10%-20-100+10+2012345Claude Opus 4.7-70%-50%-30%-10%10%-20-100+10+2012345GPT-5.4-70%-50%-30%-10%10%-20-100+10+2012345GPT-5.5-70%-50%-30%-10%10%-20-100+10+2012345Copilot minus native task resolution, percentage pointsCopilot median token change · negative means fewer tokens

1 SWE-bench Verified · 2 SWE-bench Pro · 3 SkillsBench · 4 Win-Hill · 5 Terminal-Bench 2. Axes include zero; the task-resolution axis spans ±20 points, wider than the observed range. GitHub says the quality differences are within run-to-run variance.

Back to benchmark map

03Operational and broad evidence

Terminal-Bench repeated 89 command-line tasks: the native CLI led 4 of 6 current matched pairs, but not for every model.

Published results

Swipe charts horizontally to see every label and value.

0%20%40%60%80%100%GPT-5.5 · Codex CLI83.4%GPT-5.5 · Terminus 278.2%Claude 5 Fable · Claude Code83.1%Claude 5 Fable · Terminus 280.4%Claude Opus 4.8 · Claude Code78.9%Claude Opus 4.8 · Terminus 274.6%Gemini 3.1 Pro · Gemini CLI70.7%Gemini 3.1 Pro · Terminus 270.3%Gemini 3 Pro · Terminus 274.4%Gemini 3 Pro · Gemini CLI66.3%Claude Opus 4.7 · Claude Code69.7%Claude Opus 4.7 · Terminus 266.1%

Whiskers reproduce the leaderboard's published 95% confidence intervals. The scale runs from 0 to 100%.

Back to benchmark map

04Operational and broad evidence

Artificial Analysis ran 321 tasks 3 times: the higher-scoring harness depended on the model and cost more in both full-suite comparisons.

Published results

Swipe charts horizontally to see every label and value.

Artificial Analysis performance against cost per taskTwo zero-based scatter plots compare Coding Agent Index performance and API cost while holding Claude Opus 4.7 or GPT-5.5 fixed.Claude Opus 4.7 · medium effort$0.00$0.65$1.30$1.95$2.60$3.250255075100Claude Code · 56.8% · $1.68Cursor CLI · 60.2% · $2.68OpenCode · 64.5% · $2.93GPT-5.5 · medium effort$0.00$0.65$1.30$1.95$2.60$3.250255075100Codex · 70.5% · $2.75Cursor CLI · 61.9% · $2.02Coding Agent Index, 0 to 100API cost per task in US dollars

The live page also has a GPT-5.4 pair with only 2 of the current 3 component benchmarks. It is in the downloadable observations but excluded from these full-suite panels.

Back to benchmark map

05Operational and broad evidence

Agents' Last Exam tested long professional workflows: GPT-5.5 harnesses stayed within 4.9 points, but total cost differed by more than 3 times.

Published results

Swipe charts horizontally to see every label and value.

GPT-5.5$0$550$1100$1650$2200025507510012345Claude Opus 4.7$0$550$1100$1650$22000255075100678Performance, 0 to 100%Published total evaluation cost in US dollars
PointModelHarnessResult · cost
1GPT-5.5Codex24% · $570.00
2GPT-5.5ALE-Claw23% · $326.00
3GPT-5.5OpenClaw21.1% · $477.00
4GPT-5.5Cursor20.7% · $180.00
PointModelHarnessResult · cost
5GPT-5.5Droid19.1% · $251.00
6Claude Opus 4.7Cursor20.4% · $2012.00
7Claude Opus 4.7ALE-Claw18.4% · $1164.00
8Claude Opus 4.7Claude Code13.2% · $1928.00

The panels use the source's stated 152-task public set. Sonnet 4.6 ALE-CLI rows use a different 105-task subset and remain in the downloadable data.

Back to benchmark map

06Operational and broad evidence

DrugDiscoveryBench repeated 82 expert workflows: mini-SWE-agent led the matched GPT-5.5 comparison, while Codex placed in the middle.

Published results

Swipe charts horizontally to see every label and value.

0%20%40%60%80%100%GPT-5.5 · mini-SWE-agent51.6%GPT-5.5 · Pi44.5%GPT-5.5 · Codex43.9%GPT-5.5 · OpenCode40.6%Gemini 3.5 Flash · Gemini CLI50%Gemini 3.5 Flash · Pi47%GLM 5.2 · OpenCode37.8%GLM 5.2 · Pi24.4%

Back to benchmark map

07Operational and broad evidence

Claw-SWE-Bench crossed 5 harnesses with 2 models on 350 issues: the harness spread grew from 12.5 to 27.4 points depending on the model.

Published results

Swipe charts horizontally to see every label and value.

GLM-5.1$0.00$0.63$1.25$1.88$2.50025507510012345Qwen3.6-Flash$0.00$0.63$1.25$1.88$2.500255075100678910Performance, 0 to 100%API cost per task in US dollars
PointModelHarnessResult · cost
1GLM-5.1OpenClaw73.4% · $0.79
2GLM-5.1Hermes71.1% · $0.94
3GLM-5.1ZeroClaw70.3% · $1.10
4GLM-5.1GenericAgent63.1% · $0.25
5GLM-5.1NanoBot60.9% · $2.20
PointModelHarnessResult · cost
6Qwen3.6-FlashOpenClaw66% · $0.20
7Qwen3.6-FlashHermes62.6% · $0.30
8Qwen3.6-FlashZeroClaw58.3% · $0.14
9Qwen3.6-FlashNanoBot47.4% · $0.38
10Qwen3.6-FlashGenericAgent38.6% · $0.04

Cost per task is the published configuration total divided by the common 350-task set. Every panel and axis starts at zero.

Back to benchmark map

08Operational and broad evidence

Harness-Bench ran a 6-harness by 8-model factorial: average combined score ranged from 52.4 to 76.2 across harnesses.

Published results

Swipe charts horizontally to see every label and value.

Combined score

0204060801008-model average · NanoBot76.28-model average · Hermes71.28-model average · Moltis68.88-model average · NullClaw64.48-model average · ZeroClaw61.48-model average · OpenClaw52.4

Tokens per task

0k40k80k120k160k200k8-model average · NanoBot68.7k8-model average · Hermes139.7k8-model average · Moltis134.9k8-model average · NullClaw175.1k8-model average · ZeroClaw133.2k8-model average · OpenClaw82.1k

Each bar averages the same 8-model pool. Combined score includes process judging as well as task completion.

Back to benchmark map

09Operational and broad evidence

WANDR compared 500 research runs across 2 production systems, then isolated output delivery on 45 tasks: file or share delivery scored higher in all 3 systems.

Published results

Swipe charts horizontally to see every label and value.

WANDR GPT-5.5 effort sweep Two zero-based scatter plots show soft and hard F1 against cost per task for Perplexity Search as Code and OpenAI Responses on the same 45 tasks at four effort settings. Perplexity Search as CodeOpenAI Responses Soft F1 against solver cost$0$2$4$6$80%10%20%30%40%50%12341234Hard F1 against solver cost$0$2$4$6$80%10%20%30%40%50%12341234 Solver cost per task in US dollars · 1 low · 2 medium · 3 high · 4 xhigh

Both axes start at zero. In the separate 500-task high-effort run, Search as Code scored 36.3% soft F1 at $5.20 per task; OpenAI Responses scored 12.1% at $0.50. A delivery ablation also favoured file or share delivery in all 3 systems, but used independent rollouts.

Back to benchmark map

10Operational and broad evidence

Sentry tested 86 known vulnerabilities: Pi found a similar or larger share while costing about one-quarter to one-fifth as much as Claude SDK.

Published results

Swipe charts horizontally to see every label and value.

Warden benchmark: performance and cost per task Performance from 0 to 100% plotted against cost per task from 0 to 1.5 US dollars. 0.000.300.600.901.201.50020406080100 Cost per task in US dollars Performance in % 1 2 3 4
PointModelHarnessResult · cost
1Claude Sonnet 4.6Pi29.1% · $0.23
2Claude Sonnet 4.6Claude SDK27.9% · $1.20
3Claude Opus 4.8Pi24.4% · $0.25
4Claude Opus 4.8Claude SDK19.8% · $0.93

Back to benchmark map

11Operational and broad evidence

Endor Labs tested 200 vulnerability repairs: Cursor led for Opus, but Claude Code led for Sonnet 5.

Published results

Swipe charts horizontally to see every label and value.

Functional pass rateSecurity pass rate
0%20%40%60%80%100% Claude Opus 4.7 · Cursor 79.9% 18.4% Claude Opus 4.7 · Claude Code 76.5% 16.8% Claude Opus 4.8 · Cursor 75.4% 20.7% Claude Opus 4.8 · Claude Code 73.7% 14.5% Claude Sonnet 5 · Cursor 63.1% 15.6% Claude Sonnet 5 · Claude Code 83.2% 19.6%

Back to benchmark map

12Operational and broad evidence

DeepSeek crossed one model with 8 scaffold configurations: scores spanned 65.5% to 74.2% on DeepSWE and 84.1% to 90.6% on Terminal-Bench.

Published results

Swipe charts horizontally to see every label and value.

DeepSWE v1.1

0%20%40%60%80%100%Claude Code69.8%Codex65.6%OpenCode65.5%Pi66.2%mini-SWE-agent74.2%DeepSeek Harness Minimal72.6%DeepSeek Harness Standard70.5%DeepSeek Harness PTC67.6%

Terminal-Bench 2.1

0%20%40%60%80%100%Claude Code88%Codex84.1%OpenCode85%Pi86.1%mini-SWE-agent90.3%DeepSeek Harness Minimal90.6%DeepSeek Harness Standard85.8%DeepSeek Harness PTC85.8%

Back to benchmark map

13Operational and broad evidence

Browser Use ran concurrent revision comparisons: the candidate gained 7.53 points on 60 Luna tasks and tied at 84 of 106 on Hard.

Published results

Swipe charts horizontally to see every label and value.

BU Bench V2

0%20%40%60%80%100%Candidate 29e2b5e61.8%Reference c3f7fac54.2%

Internal Bench Hard

0%20%40%60%80%100%Candidate 29e2b5e79.2%Reference 769ea1479.2%

The Luna candidate-reference difference has a published paired 95% bootstrap interval of +0.10 to +15.42 points. The Hard reference has 105 actual judgments for 106 assigned tasks, so that pair is excluded from matched-pair counts.

Back to benchmark map

14Focused domain evidence

Harvey tested 180 legal tasks: the Applied Compute harness ranged from 0.9 points worse to 23 points better across 6 fixed models.

Published results

Swipe charts horizontally to see every label and value.

0%20%40%60%80%100%GLM 5.1 · Baseline LAB86.2%GLM 5.1 · Applied Compute85.3%Claude Opus 4.8 · Baseline LAB88%Claude Opus 4.8 · Applied Compute91.1%GPT-5.5 · Baseline LAB84.9%GPT-5.5 · Applied Compute88.5%Claude Sonnet 4.6 · Baseline LAB65.5%Claude Sonnet 4.6 · Applied Compute88.5%GPT-5 Mini · Baseline LAB64.3%GPT-5 Mini · Applied Compute68.9%Kimi K2.6 · Baseline LAB82.5%Kimi K2.6 · Applied Compute82.5%

Whiskers reproduce the source's published error values. Harvey does not define them as confidence intervals. The scale runs from 0 to 100%.

Back to benchmark map

15Focused domain evidence

LAB held DeepSeek V4 Pro fixed on 100 legal tasks: changing the harness moved pooled criterion pass rate from 3.5% to 80.1%.

Published results

Swipe charts horizontally to see every label and value.

0%20%40%60%80%100%DeepSeek V4 Pro · Optimized LAB80.1%DeepSeek V4 Pro · Vanilla LAB63.4%DeepSeek V4 Pro · Pi45.4%DeepSeek V4 Pro · Goose23.2%DeepSeek V4 Pro · mini-swe-agent3.50%

Back to benchmark map

16Focused domain evidence

LatchBio repeated 121 biology tasks: Pi's point estimate was 4.6 points higher on shorter work, while Claude Code's was 4.8 points higher on long workflows; intervals overlapped.

Published results

Swipe charts horizontally to see every label and value.

0%20%40%60%80%100%scBench-Long · Claude Opus 4.8 · Claude Code25.4%scBench-Long · Claude Opus 4.8 · Pi20.6%scBench-Long · GPT-5.5 · Codex15.9%scBench-Long · GPT-5.5 · Pi15.9%TxBench-PP · Claude Opus 4.8 · Pi59.3%TxBench-PP · Claude Opus 4.8 · Claude Code54.7%TxBench-PP · GPT-5.5 · Pi55.3%TxBench-PP · GPT-5.5 · Codex47.3%

Back to benchmark map

17Focused domain evidence

CSI ran 5 scaffolds with one model on 33 security challenges: Claude and Codex tied on solves, while Codex cost one-third as much.

Published results

Swipe charts horizontally to see every label and value.

CSI scaffold comparison: performance and cost per task Performance from 0 to 100% plotted against cost per task from 0 to 200 US dollars. 04080120160200020406080100 Cost per task in US dollars Performance in % 1 2 3 4 5
PointModelHarnessResult · cost
1alias2-miniCSI::Claude45.5% · $155.21
2alias2-miniCSI::Codex45.5% · $51.91
3alias2-miniCSI::Mistral30.3% · $29.39
4alias2-miniCSI::GCAI30.3% · $38.76
5alias2-miniCSI::CAI21.2% · $22.03

Back to benchmark map

18Focused domain evidence

XOR tested 136 vulnerability repairs: Claude Code and Codex beat OpenCode on all 3 published matched-model pairs.

Published results

Swipe charts horizontally to see every label and value.

Vulnerability-Agent-Bench: performance and cost per task Performance from 0 to 100% plotted against cost per task from 0 to 25 US dollars. 0510152025020406080100 Cost per task in US dollars Performance in % 1 2 3 4 5 6
PointModelHarnessResult · cost
1Claude Opus 4.6Claude Code56.6% · $1.66
2Claude Opus 4.6OpenCode42.6% · $22.13
3GPT-5.2Codex58.1% · $3.08
4GPT-5.2OpenCode46.3% · $3.08
5GPT-5.2-CodexCodex46.3% · $3.08
6GPT-5.2-CodexOpenCode35.3% · $3.08

Back to benchmark map

19Focused domain evidence

WildClawBench tested 60 long-horizon tasks: changing the harness produced double-digit score shifts and changed the winner by model.

Published results

Swipe charts horizontally to see every label and value.

WildClawBench: performance and cost per task Performance from 0 to 100% plotted against cost per task from 0 to 0.7 US dollars. 0.000.140.280.420.560.70020406080100 Cost per task in US dollars Performance in % 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
PointModelHarnessResult · cost
1GPT-5.4OpenClaw50.3% · $0.33
2GPT-5.4Claude Code48.4% · $0.61
3GPT-5.4Codex56.8% · $0.57
4GPT-5.4Hermes Agent50.7% · $0.44
5GLM 5OpenClaw42.6% · $0.19
6GLM 5Claude Code31% · $0.21
7GLM 5Codex38.9% · $0.13
8GLM 5Hermes Agent46.4% · $0.44
PointModelHarnessResult · cost
9MiMo V2 ProOpenClaw40.2% · $0.44
10MiMo V2 ProClaude Code29.9% · $0.15
11MiMo V2 ProCodex35.3% · $0.15
12MiMo V2 ProHermes Agent48.1% · $0.26
13MiniMax M2.7OpenClaw33.8% · $0.12
14MiniMax M2.7Claude Code32% · $0.09
15MiniMax M2.7Codex35.8% · $0.06
16MiniMax M2.7Hermes Agent37.1% · $0.11

Back to benchmark map

20Focused domain evidence

CORE-Bench Hard tested 45 scientific reproductions: Claude Code more than doubled verified success while costing less than CORE-Agent.

Published results

Swipe charts horizontally to see every label and value.

CORE-Bench Hard: performance and cost per task Performance from 0 to 100% plotted against cost per task from 0 to 4 US dollars. 0.000.801.602.403.204.00020406080100 Cost per task in US dollars Performance in % 1 2
PointModelHarnessResult · cost
1Claude Opus 4.5Claude Code77.8% · $1.94
2Claude Opus 4.5CORE-Agent42.2% · $3.39

Back to benchmark map

21Focused domain evidence

Prime Intellect compared 3 models at high effort across 9 long-context evaluations: Prime Agent led 20 of 27 point estimates.

Published results

Swipe charts horizontally to see every label and value.

EvaluationPrime · GLM-5.2Pi-mono · GLM-5.2Prime · Opus 5Claude Code · Opus 5Prime · GPT-5.6 SolCodex · GPT-5.6 Sol
OOLONG Yahoo70%42%90%92%94%90%
OOLONG-Pairs87.4%55.6%92.9%92.2%91.1%89.5%
OBLIQ-Bench nDCG@1066.9%63.5%80.2%79.5%61.2%64.6%
LongBench Pro English77.7%76.8%80.4%79%79.4%79%
LongBench v268%69.6%74.4%74.6%71.4%70.4%
ManyIH Coding42.4%38.6%53.6%52.2%49.9%45.4%
ManyIH IF20.9%16.4%22.5%17.5%21.6%23.2%
LongCoT-Mini63.8%61.3%72.2%55.8%67.1%68.1%
EmulatorBench verifier20.8%0%4.7%6.2%27.5%22.8%

All columns use the source's high reasoning label. Values are shown on a 0 to 100 scale. Metrics differ by row, and the source publishes no uncertainty intervals.

Back to benchmark map

22Small and diagnostic evidence

The Scaffold Effect ran 300 trials: Goose used about 41 times fewer reported tokens than OpenCode, with pass rates within 2 to 8 points.

Published results

Swipe charts horizontally to see every label and value.

Pass rate

0%20%40%60%80%100%Qwen 3.6 Plus · Goose48%Qwen 3.6 Plus · OpenCode50%Qwen 3.6 Plus · OpenHands-SDK50%MiniMax M2.5 · Goose38%MiniMax M2.5 · OpenCode46%MiniMax M2.5 · OpenHands-SDK46%

Reported tokens per solved task

0m0.3m0.6m1m1.3m1.6mQwen 3.6 Plus · Goose0.03mQwen 3.6 Plus · OpenCode1.15mQwen 3.6 Plus · OpenHands-SDK0.84mMiniMax M2.5 · Goose0.04mMiniMax M2.5 · OpenCode1.55mMiniMax M2.5 · OpenHands-SDK0.84m

Both scales start at zero. Token accounting is harness-reported and incomplete, so treat the efficiency contrast as diagnostic.

Back to benchmark map

23Small and diagnostic evidence

HAL compared 15 models on 50 SWE-bench tasks: SWE-Agent usually solved more, while HAL Generalist usually cost less.

Published results

Swipe charts horizontally to see every label and value.

0%20%40%60%80%100%Claude Sonnet 4.5 · highSWE 72% · $9.28HAL 40% · $1.92Claude Opus 4.1 · highSWE 54% · $32.00HAL 46% · $8.00o4-mini · lowSWE 54% · $5.18HAL 6% · $1.74GPT-5 · mediumSWE 46% · $3.26HAL 12% · $1.15Gemini 2 Flash · defaultSWE 24% · $0.09HAL 2% · $0.15DeepSeek R1 · defaultSWE 0% · $0.08HAL 6% · $2.93

Six representative pairs are shown on a zero-to-100 performance scale; all 15 pairs are in the downloadable observations. Dollar labels are published total cost divided by 50 tasks.

Back to benchmark map

24Small and diagnostic evidence

HarnessBench tested 27 recent bug fixes: several configurations finished within one task of the leader, but runtime differed more clearly.

Published results

Swipe charts horizontally to see every label and value.

0%20%40%60%80%100%GPT-5.5 · Codex81.5%GPT-5.5 · Codex77.8%GPT-5.5 · Cursor77.8%Claude Opus 4.7 · Cursor77.8%

Back to benchmark map

25Small and diagnostic evidence

A-CODE-CLI repeated 10 web scenarios: OpenCode had a 2.4-point backend lead and cost 44% less than Claude Code.

Published results

Swipe charts horizontally to see every label and value.

A-CODE-CLI: performance and cost per task Performance from 0 to 100% plotted against cost per task from 0 to 2.5 US dollars. 0.000.501.001.502.002.50020406080100 Cost per task in US dollars Performance in % 1 2
PointModelHarnessResult · cost
1Claude Sonnet 4.6OpenCode77.3% · $1.03
2Claude Sonnet 4.6Claude Code74.9% · $1.83

Back to benchmark map

26Small and diagnostic evidence

OpenBench repeated 3 coding tasks: every harness passed, so the useful difference was efficiency rather than correctness.

Published results

Swipe charts horizontally to see every label and value.

Mean wall time

0s40s80s120s160s200sGPT-5.5 · Pi45.6sGPT-5.5 · Cursor48.8sGPT-5.5 · Codex108.4sGPT-5.5 · OpenCode175.2s

Tokens per solve

020000400006000080000100000GPT-5.5 · Pi17494GPT-5.5 · Cursor21370GPT-5.5 · OpenCode44472GPT-5.5 · Codex75778

Back to benchmark map

27Small and diagnostic evidence

Portkey measured 2 trivial requests: fixed input overhead ranged from about 2,600 tokens for Pi to 27,000 for Claude Code.

Published results

Swipe charts horizontally to see every label and value.

0600012000180002400030000Pi · Pi2600Codex · Codex15000Claude Code · Claude Code27000

This diagnostic measures fixed input overhead. It does not measure coding quality.

Back to benchmark map

Evidence matrix

26 studies contain at least one matched-model harness comparison. Portkey is an efficiency diagnostic, not a quality benchmark.

Dates are the publication or public-release date of the comparison used here. For live leaderboards, they mark when the referenced comparison first appeared.

No.StudyTask surfaceModel fixedIncludes PiRepeatsCost
01Databricks coding-agent benchmarkRecent human-written pull requests from a multi-million-line internal codebaseyesyesnot clearyes
02GitHub Copilot harness evaluationFive coding and terminal benchmark suites, including SWE-bench and Terminal-Bench 2yesnoyesyes
03Terminal-Bench 2.1Containerized command-line tasks with deterministic final-state checksyesnoyesyes
04Coding Agent Index321 tasks across DeepSWE, Terminal-Bench v2 and SWE-Atlas-QnAyesnoyesyes
05Agents' Last ExamLong-horizon professional workflows across CLI, GUI, files, applications and web researchyesnonot clearyes
06DrugDiscoveryBenchExpert-curated drug-discovery and life-sciences workflowsyesyesyesyes
07Claw-SWE-BenchMultilingual GitHub issue-resolution tasksyesnonoyes
08Harness-Bench paperExecutable office, retrieval, analytics, SRE and software-engineering workflowsyesnonono
09WANDR system comparison and delivery ablationWide-and-deep web research requiring large, evidence-backed collectionsyesnonoyes
10Warden benchmarkKnown-vulnerability recognition in files tied to historical Sentry vulnerabilitiesyesyesnoyes
11Agent Security LeagueReal vulnerability-fixing tasks with hidden functional and security testsyesnonono
12DeepSeek V4.1 Flash scaffold evaluationRepository issue resolution and containerized command-line tasksyesyesyesno
13Browser Use Pi concurrent revision comparisonLive browser research tasks and an internal hard browser benchmarkyesyesnoyes
14Harvey LAB harness ablationHeld-out legal workflows scored against expert-written criteriayesnonono
15Harness optimization on LABRealistic legal matters requiring evidence-backed deliverablesyesyesnoyes
16LatchBio behavioural-prior studyShort-horizon preclinical pharmacology and long biological analysis workflowsyesyesyesyes
17CSI scaffold comparisonCyBench capture-the-flag security challengesyesnonoyes
18Vulnerability-Agent-BenchReal vulnerability repairs in isolated containersyesnonoyes
19WildClawBenchBilingual, multimodal and long-horizon agent tasksyesnonot clearyes
20CORE-Bench HardScientific reproducibility tasksyesnonoyes
21Prime Agent long-context comparisonNine long-context reasoning, instruction-following and emulator-construction evaluationsyesyesnot clearno
22The Scaffold EffectA 50-task slice of Terminal-Bench Proyesnonono
23HAL SWE-bench Verified MiniA fixed 50-task subset of SWE-bench Verifiedyesnonoyes
24HarnessBenchRecent bug-fix issues from 9 open-source repositoriesyesnonoyes
25A-CODE-CLITen full-stack web development scenariosyesnoyesyes
26OpenBench M4.5Three repository coding tasks with executable checkersyesyesyesno
27Harness TaxTwo trivial prompts routed through a gatewaynoyesnono

How to read harness comparisons · Download observations · Download study records

Screened but not counted

These sources are relevant to the search, but they do not currently isolate a harness effect or they duplicate an included benchmark family.

SourceDateDispositionReason
Factory Terminal-Bench article25 Sept 2025Not counted separatelyThe headline Droid comparisons use different model variants from Claude Code and Codex. Valid Terminal-Bench comparisons are covered under the benchmark family.
Harvey Legal Agent Benchmark introduction6 MayContext onlyThe launch defines the legal benchmark but publishes no comparison that changes the harness while holding the model fixed.
Harvey LAB initial results26 MayExcludedIt compares models under a fixed harness. The later 22 June harness ablation is included instead.
FrontierSWEApr 2026WatchlistThe benchmark repeats 17 long-horizon tasks, but each model currently appears under only one harness, so the harness effect cannot be isolated.
Artificial Analysis general agents catalogueNot datedExcludedThis is a product catalogue, not a benchmark. Alternate Coding Agent Index URLs are merged into the single included study.
SWE-bench Pro harness-model pairing comparison7 AugWatchlistThe 10-harness, 2-model comparison isolates harness effects across 250 tasks, but the post links no raw table or repository and publishes only selected exact values. Add observation rows when the underlying data is public.
FrontierHarness Eval v11 SeptWatchlistThe study holds Kimi K3 and 30 task identities fixed across 12 configurations, but it does not publish a shared reasoning effort, temperature, context limit or complete invocation manifest. Its effective cost per pass divides total known task cost by successful tasks, despite repeated median labelling, and 17 failed cells lack normalized cost.
Prime Agent ARC-AGI-3 comparison24 AugContext onlyPrime Agent's 95.5% RHAE result is reproducible from public runner artefacts, but the 30.2%, 38.3% and other comparison points come from external runs with unmatched prompts, budgets, effort settings and protocols. They cannot isolate a harness effect.
Browser Use Pi historical cross-harness comparisons9 SeptContext onlyThe historical BrowserCode and Python Browser Harness cohorts share task IDs, model labels and effort labels with Browser Use Pi, but they ran on different dates, runners, browser dimensions and live website states. The included study uses the stronger concurrent revision pair instead.
OpenAI ARC-AGI-3 harness ablation30 JulWatchlistOpenAI reports that retained reasoning and context compaction raised GPT-5.6 Sol's public-set score from about 13.3% to 38.3% while reducing output tokens, but the thread does not publish enough shared effort, budget and run-level data for strict inclusion.

Data and source access

External datasets remain under their publishers' terms. This repository stores derived observations and links to the original data.

SourceAccessLicence noteLinks
Benchmarking coding agents on Databricks' multi-million-line codebaseNo public task-level dataset foundNot stated for benchmark dataResults
Warden benchmarkPublic aggregate tableNot stated for benchmark resultsResults · Dataset or repository
LatchBio behavioural-prior studyPublic TxBench-PP and scBench-Long JSON summaries and trajectories; aggregate prompt-ablation table in the articleCheck the linked benchmark repositories and dataset termsResults · Dataset 1 · Dataset 2
Harness optimization on the Legal Agent BenchmarkPublic article source, harness code, frozen split, dev and test results, and proposer tracesArticle text and diagrams are CC BY 4.0; check the linked code and benchmark termsResults · Dataset or repository
OpenBenchPublic repository with committed data for several milestonesMITResults · Dataset or repository
DrugDiscoveryBenchTasks and code are public; rubrics and reference answers are gated on Hugging FaceMIT for repository code and tasks; separate terms apply to bundled data and gated materialResults · Dataset or repository
Artificial Analysis Coding Agent IndexLive public result page and methodology; no stable bulk dataset foundResult-data licence not stated; site terms applyResults
Endor Labs Agent Security LeaguePublic leaderboard and articlesNot stated for benchmark dataResults
WANDR benchmark and output-delivery ablationPublic paper, 500 task packages, evaluation code and aggregate tables; evaluated run artefacts are not includedApache 2.0 repository and CC BY 4.0 paper; third-party task sources retain their own termsResults · Dataset or repository
Vulnerability-Agent-BenchPublic explorer and linked Weights and Biases projectNot stated for benchmark dataResults · Dataset or repository
WildClawBenchPublic code, tasks, runtime images and datasetCheck the repository and dataset cardsResults · Dataset or repository
HarnessBenchPublic code and resultsCheck the repository licenceResults · Dataset or repository
CORE-Bench HardPublic harness, leaderboard and encrypted tracesCheck the repository licenceResults · Dataset or repository
A-CODE-CLIPublic report and one example taskCheck the example repository licenceResults · Dataset or repository
The harness taxPublic article with aggregate valuesNot stated for request logsResults
GitHub Copilot agentic harness evaluationPublic methods and aggregate charts; no raw result rows foundNot stated for benchmark result dataResults
Terminal-Bench 2.1Public leaderboard, paper, experiments and submissionsApache-2.0 repositories; paper CC BY 4.0Results · Dataset or repository
Agents' Last ExamPublic paper, HTML tables, code and dataData and paper CC BY 4.0; software Apache-2.0Results · Dataset or repository
Claw-SWE-BenchPublic paper, adapters, dataset and result artefactsMIT repository; check dataset card for task data termsResults · Dataset or repository
Harness-Bench: Measuring Harness Effects on LLM AgentsPublic paper, tasks, code and aggregated JSONPaper CC BY 4.0; repository has no licence file as capturedResults · Dataset or repository
Harvey LAB harness ablation with Applied ComputePublic aggregate chart; baseline benchmark and harness are public, ablation split is notMIT for public LAB repository; ablation result-data licence not statedResults · Dataset or repository
Cybersecurity scaffold comparisonPublic paper and project page; no dedicated experiment artefact foundPaper CC0; related code has mixed MIT and research-only termsResults
The Scaffold EffectPublic raw results, prompts and analysisMIT repositoryResults · Dataset or repository
HAL SWE-bench Verified MiniPublic leaderboard and downloadable tracesArchived trace repository has no licence file as capturedResults · Dataset or repository
DeepSeek V4.1 Flash scaffold evaluationPublic model card, technical report, evaluation guide and selected scaffold code; no task-level evaluated results or trajectoriesCheck the model, harness and benchmark repositories for their separate termsResults · Dataset 1 · Dataset 2
Prime Agent technical reportPublic paper and open-source harness; included long-context values are aggregate-onlyMIT for Prime Agent code; check each benchmark for its data termsResults · Dataset 1 · Dataset 2 · Dataset 3
Browser Use Pi benchmark evidencePublic aggregate reports, configuration manifests, task-level scores, costs and selected audits; tasks and most traces need organization accessMIT repository; private benchmark tasks retain their own access termsResults · Dataset or repository