Harbor Adapters and Harbor-Index: Scaling and Curating Agent Evaluation
Agent evaluation is fragmented. One benchmark requires a particular software environment, another a custom tool interface, and another its own verifier and scoring logic. Supporting a new agent often means rebuilding those connections, making broad evaluation expensive and cumbersome.
Harbor Adapters provide a shared interface that makes broader evaluation practical. That evaluation, together with subsequent task-quality review, informs Harbor-Index: a set of 82 tasks across 29 benchmarks, selected to be compact, diverse, difficult, and audited.
Harbor Adapters and Harbor-Index address connected questions: how can agents be evaluated across heterogeneous benchmarks, and what should a smaller evaluation set retain? Answering the second requires both the broad comparisons enabled by Harbor Adapters and closer inspection of what passing and failing mean.
1. A Shared Interface Makes Broad Evaluation Practical
Supporting m benchmarks and n agents can require O(mn) separate integrations. Harbor Adapters reduce that burden to O(m+n): each benchmark translates its tasks into Harbor’s shared schema of instruction, environment, tests, and solution, and each agent integrates with Harbor once. Benchmark-specific execution and scoring remain behind that interface.
As of the paper’s May 2026 snapshot, Harbor Adapters support more than 80 benchmarks and 22 agents. Adapters undergo automated and human review, with parity experiments comparing original and adapted implementations under matched configurations. These checks support adaptation fidelity; they do not independently establish that every underlying task or verifier is valid.
The infrastructure enables an evaluation spanning 54 benchmarks, 6,627 tasks, and eight models, each running with Terminus-2 and its vendor-native harness. Across 16 model–harness configurations and three trials per setting, the study consumes 226 billion tokens and more than $300,000 in compute. That scale makes cross-benchmark patterns observable, while its expense makes understanding the value of each additional measurement consequential.
Making broad evaluation practical also raises a question of selection: what should a compact evaluation retain from that wider view? Harbor-Index is the smaller, audited collection informed by the broad study and subsequent task review. Its composition below previews the outcome; understanding the choices behind it requires examining both what broad evaluation reveals and what task review adds.
2. What Becomes Visible at Scale
Progress is uneven across the benchmark landscape. The paper identifies 13 of the 54 benchmarks as largely saturated, with state-of-the-art scores above 90%. Functional and competitive-coding tasks are approaching saturation, repository-level feature implementation and issue fixing are progressing quickly, while software-speedup tasks remain challenging. A broad evaluation includes both familiar tasks with little headroom and problems that agents still struggle to solve.
This progress is a property of the benchmark landscape as a whole; it does not by itself say what separates one evaluated system from another. Because every model in this study runs under both a common cross-family harness and its own native harness, the same evaluation also supports a more targeted comparison: how score differences associated with model choice compare with those associated with harness choice.
2.1 Model Differences Dominate in This Comparison
After accounting for benchmark difficulty, the fitted analysis finds larger model effects than harness effects. The fitted model-effect range is 5.2 times the fitted harness-effect range across the evaluated configurations.
This finding is specific to the study’s comparison. Each model uses Terminus-2 and its vendor-native harness, rather than every model being tested with every harness. Harness design still affects tool access and execution behavior; the result does not establish a universal hierarchy between model capability and agent scaffolding.
2.2 Fewer Tokens Do Not Mean Lower Cost
Model choice affects more than benchmark scores. In this study, the frontier-model group used fewer tokens, yet still cost more per trial. Token usage, dollar cost, and task success therefore capture different aspects of evaluation efficiency; none alone describes the tradeoff.
Empirical difficulty is one minus the average score across the evaluated configurations, not an intrinsic task property. These comparisons help assess the resources spent on a given task set. A separate question is how much distinct information that set provides, regardless of how many benchmarks it draws from.
2.3 More Benchmarks Do Not Always Add More Signal
Many benchmarks rank the evaluated systems similarly. Much of the variation across their scores is concentrated in a few shared components, suggesting that additional benchmarks do not always contribute proportionate new information.
A benchmark-selection analysis makes that overlap concrete. After 12 benchmarks are greedily selected, every remaining benchmark has an absolute Spearman rank correlation of |ρ| ≥ 0.7 with at least one selected benchmark.
Overlap also appears within benchmarks: carefully selected small task subsets can recover much of the system ranking produced by the full benchmark.
These subsets were selected retrospectively using full-benchmark results; their ranking agreement should not be assumed to generalize to arbitrarily chosen subsets or to future model populations. Overlapping signal does not make benchmarks or tasks interchangeable. It does provide a reason to consider a more selective evaluation set, but ranking correlation alone cannot determine which tasks deserve a place.
3. Difficulty Is Not the Same as Quality
Selecting an informative evaluation set requires more than identifying tasks with low scores. Some failures reflect incorrect reasoning or implementation; others arise because the task cannot execute correctly or the verifier demands something the instructions never specified. Among the hardest candidates examined during human review across more than 30 benchmarks, roughly one third were rejected as broken rather than genuinely difficult. That finding concerns selected hard candidates, not the entire evaluation pool.
Apparent difficulty can come from an environment that never reaches a gradable state, or a verifier that demands an answer format the task never specified. In four audited tasks from GAIA2, a simulated-environment benchmark, an adapter defect prevented the events required for grading from firing. A long-context reasoning task from AA-LCR exposed a different problem: it rejected “30.4 percentage points” while accepting “0.304,” enforcing an unstated representation requirement.
Such defects can produce false negatives, rejecting valid work. Verifier loopholes create the opposite problem: exposed reference answers or shared-environment tampering can allow an agent to pass without solving the intended task. Neither kind of error is ruled out simply by reproducing an existing benchmark’s behavior.
The analyzed failure trajectories also reveal substantive agent errors, including wrong factual answers, algorithmic bugs, and hidden-test regressions, across the evaluated frontier models and harnesses. Task selection must distinguish these failures from defects in the evaluation itself. Together, these observations explain Harbor-Index’s emphasis on compactness and auditing: overlapping ranking signal motivates a smaller collection, while task defects make low scores insufficient grounds for inclusion.
4. Harbor-Index: Turning Broad Evaluation Into a Smaller, Audited Set
Keeping Harbor-Index compact requires selection; preserving diversity and challenge requires attention to what those tasks demand; auditing checks whether their instructions, environments, and verifiers support the intended work. The broad evaluation informs that selection; it does not establish that the resulting set statistically preserves the full suite’s rankings. The curation process therefore combines performance-based filtering with task review, repair, and re-evaluation.
4.1 Low Pass Rates Are Only the Starting Point
The curation funnel begins by narrowing the initial pool of 6,627 tasks to 1,311 on difficulty alone, retaining only tasks with at most six successes across 18 trials from three frontier models, each under two harnesses.
A Gemini-3-Flash auditor, followed by a pool of 14 domain-experienced human reviewers, checks both whether instructions and verification align and whether difficulty comes from substantive reasoning, domain expertise, or execution demands. This narrows the pool to 307 candidates, then to more than 110, before a senior panel selects 100 based on difficulty, diversity, and insights from model behavior for a final, trajectory-grounded audit. That last stage repairs fixable defects, reruns the filtering models, and drops any task that remains broken or turns out too easy once repaired. Correcting a defect can change a task's apparent difficulty, so a low pass rate should not preserve its place once that source of difficulty is gone. What remains is the final 82-task release.
4.2 How Frontier Agents Fare on Harbor-Index
GPT-5.5 with Codex reaches 28.0%, the highest reported pass rate; no evaluated configuration exceeds 30%. Separate from the earlier broad study, this Harbor-Index evaluation covers nine models, 18 model–harness configurations, and 82 tasks, producing 1,476 rollouts. For continuous-score tasks, Harbor-Index “SOTA-thresholds” outcomes into pass/fail using fixed task-specific thresholds: solutions must exceed the threshold or, for bounded metrics, reach the metric’s upper bound.
Higher pass rates can come with higher evaluation costs, but both depend on the setup: harnesses, tool access, timeouts, and scoring conditions affect outcomes, while provider pricing and cache behavior affect expense. Several open-weight models under Terminus-2 sit on this cost–performance Pareto frontier, trading pass rate for a substantially lower cost per run. Low pass rates show remaining challenge under these conditions, but failures can reflect both model limitations and constraints imposed by the harness and available tools.
5. Toward More Reliable and Informative Agent Evaluation
Shared infrastructure makes broad evaluation practical. That broader view reveals differences among systems, the costs of stronger performance, overlapping ranking signal, and defects that complicate score interpretation. Harbor-Index turns those lessons into a compact, diverse, difficult, and audited testbed.
The findings remain bounded by the models, harnesses, and execution settings evaluated. LLM judges can introduce error, and Harbor-Index is a proxy for a broader task distribution rather than a universal measure of agent capability. Future systems may saturate tasks or discover new exploits, making continued auditing and maintenance necessary.