Quantifying infrastructure noise in agentic coding evals

Key Actionable Insights

1

Specify both a guaranteed allocation (floor) and a hard kill threshold (ceiling) for container resources in agentic evals, rather than a single pinned value. Setting them equal leaves zero headroom, causing transient memory spikes to OOM-kill containers that would otherwise succeed, artificially depressing scores.

This is the primary recommendation from the article. Terminal-Bench 2.0's Kubernetes implementation initially set both parameters equal, resulting in 5.8% infra error rates that dropped to 2.1% with a 3x ceiling.

2

Calibrate the resource headroom band so that scores at the floor and ceiling of the band fall within statistical noise of each other. For Terminal-Bench 2.0, a 3x multiplier over per-task specs achieved this balance, cutting infra errors by two-thirds while keeping score differences non-significant (p = 0.40).

The exact multiplier will vary by benchmark and task distribution, so it should be empirically determined and reported. The key principle is that the headroom should eliminate infrastructure artifacts without making tasks easier to solve.

3

Treat resource configuration as a first-class experimental variable in agentic evaluations, documented and controlled with the same rigor as prompt format or sampling temperature. Without published setup configurations, external observers cannot distinguish genuine capability differences from infrastructure artifacts.

This applies to AI labs, benchmark maintainers, and anyone publishing or consuming benchmark results. Publishing recommended specs (as Terminal-Bench 2.0 does) helps, while specifying enforcement methodology would close the remaining gap.

4

Apply skepticism to agentic coding benchmark leaderboard differences below 3 percentage points. The observed spread across moderate resource configurations is just below 2 percentage points, and naive binomial confidence intervals already span 1-2 percentage points—infrastructure confounders stack on top of that.

This is critical for decision-makers choosing which models to deploy based on benchmark rankings. A few-point lead might reflect a genuine capability difference or simply reflect different infrastructure configurations.

5

Run public coding evals at multiple times and on multiple days to average out temporal variance. Pass rates can fluctuate with time of day due to API latency varying with traffic patterns, adding another layer of noise beyond resource configuration.

This addresses confounders that are hard to control even with standardized resource specs. External evaluators who cannot dedicate hardware are particularly vulnerable to this variance.

6

Be aware that resource configuration changes what an agentic eval actually measures: tight limits reward efficient, lean coding strategies while generous limits reward agents that exploit available resources with heavyweight tools. Collapsing both into a single score without specifying resources makes results hard to interpret.

Both measurement modes are legitimate depending on the use case—but they should be explicitly distinguished. Different deployment scenarios may favor different strategies.