Model leaderboards hide the harness tax

Model leaderboards hide the harness tax

Suppose you are choosing a coding agent for an internal release lane. Two candidates both resolve 46% of your trial tasks. The procurement sheet calls them tied and moves to model price.

Then operations reviews the traces. One candidate sends far more tokens through each solved task, waits through idle turns that do not edit or execute anything, and tends to die at the same verification boundary. The other reaches the same pass rate with shorter active trajectories. Engineers supervising the first candidate spend more time checking whether repeated reads and empty turns are harmless. The pass-rate tie was real; it was also an incomplete release decision.

The operational unit of comparison is not a model name. It is a model plus harness plus environment plus budget. The harness chooses context, exposes tools, serializes observations, retries, and stops. The environment determines available state and execution conditions. The budget decides how much search, waiting, and repeated inference the system may buy. Change any of these and the label on the model row no longer identifies the system you measured.

The empirical evidence here needs prominent bounds. The Scaffold Effect v1↗ is preliminary work on two models—Qwen 3.6 Plus and MiniMax M2.5—across three open-source harnesses—Goose, OpenCode, and OpenHands-SDK—on a stratified 50-task subset of Terminal-Bench Pro. It is not a survey of all models, harnesses, coding work, or production deployments.

Within that small study, harness choice produced up to a 40-times difference in tokens per solved task. Paired within-model pass-rate differences were 0–8 percentage points; the 95% paired-task bootstrap intervals included zero except for the largest gap. Harness-specific failure fingerprints replicated across both models: the paper associates Goose with reasoning failures, OpenHands-SDK with verification and maximum-turn failures, and OpenCode with idle loops and time failures. Those findings are suggestive precisely because the pass-rate differences were modest while the operating behavior diverged.

The 46% in the opening is not invented: in one MiniMax cell, OpenCode and OpenHands-SDK both reported that pass rate. The paper reports different token totals per solved task for those harnesses, and its broader trace analysis surfaces no-action turns and repeatable failure mixes. Turning those measurements into expected developer interruption or oversight cost is an engineering inference, not a directly measured labor study. A procurement team should measure those costs in its own lane rather than price them from this preprint.

AgentCompass v3↗ provides architecture evidence for keeping the variables separate. Its evaluation infrastructure defines independent Benchmark, Harness, and Environment components: benchmark owns task material and scoring, harness owns the interactive agent logic, and environment owns execution primitives and isolation. That modular design is not performance evidence for any configuration. It simply demonstrates that the separation can be made concrete in evaluation software.

The older AI Agents That Matter↗ supplies a broader scholarly framing. It argues for cost-controlled agent evaluation, distinguishes model-development questions from downstream procurement questions, and documents reproducibility problems when agent evaluations are weakly standardized. Its lesson for a release review is not “pick the cheapest row.” It is to compare systems on the dimensions the downstream decision actually bears and to publish enough configuration detail that another team can reproduce the comparison.

So the release table needs columns beyond pass rate: model and harness versions, environment image, task set, token and wall-clock budgets, tokens per verified outcome, latency distribution, idle-turn rate, failure class, and human interventions. A higher-cost configuration may still be the right choice if it buys a material reliability or risk improvement. A tie may conceal an obvious operational loser. Either way, the trade is visible.

There is a fair countercase. A model-only comparison is valid when harness, environment, task set, evaluator, and budget are fixed and fully matched. Under those controls, changing only the model answers a real question about the model's contribution. The error is carrying that conclusion into a deployment whose surrounding system differs.

What this does not prove: The Scaffold Effect v1 does not establish a universal 40-times harness tax, rank commercial products, or show that its failure fingerprints persist beyond two models and three harnesses on 50 Terminal-Bench Pro tasks. AgentCompass does not prove modularity improves scores, and AI Agents That Matter does not supply current procurement prices. The claim is narrower: a model name alone does not identify the measured operating system when harness, environment, or budget varies.

Which hidden harness cost would reverse your current release decision even if the headline pass rate stayed exactly the same?