The harness is the capability multiplier

A model can do impressive work at every step and still fail the task. It can decompose around the wrong invariant, lose a key fact at a handoff, verify an answer only against the assumptions that produced it, stop after writing files that do not survive a cold start, or reach a timeout with the correct plan still in its head.

A model supplies local capability. The harness decides where to apply it, what context crosses each handoff, what can run in parallel, and what must be verified before the result ships. Those choices matter more as work stretches from one response into hours of research, implementation, and repair.

We built two related systems around this idea:

1. /mission for Claude Code and Codex turns an ambitious goal into a durable contract, an adaptive task graph, routed workers, and a role-separated acceptance stage.
2. Medley Bench is the benchmark-specialized form of the same architecture. Before evaluation, an unsupervised meta-harness uses a setup agent to select tasks, study the resulting execution traces, and edit the planning, worker, acceptance, and runtime skills for that workload.

The two systems were designed together. The mission harness organizes work within one solve. The meta-harness learns a better operating procedure across solves. Each benchmark gets one setup campaign with one setup-agent model class; the frozen procedure is then reused—not retuned—across frontier, previous-generation, and open-weight model/runtime arms. Neither system changes model weights.

The learning loop is not benchmark-only. With operator approval, completed missions can become unlabeled traces: the system self-evaluates each role, proposes bounded skill edits, and carries approved lessons into later missions without a grader, labeled data, or a weight update.

The graph changes the frontier

Medley produced the highest reported result on four public benchmarks. Figure 1 compares each result with the strongest comparable public score we found.

Figure 1. Each panel uses the benchmark's native 0–100 metric; values are comparable within a panel, not across benchmarks. Exact configurations and sources appear in the table below.

The table below gives the exact metric and model-family context for each comparison.

Medley uses the strongest reported model in each row as its primary worker: GPT-5.6 Sol on Terminal-Bench 2.1, Fable 5 on HealthBench Professional, Opus 4.8 on ViBench, and GPT-5.5 on DrugDiscoveryBench. It adds planning and acceptance around that worker; the worker itself is not upgraded between the comparator and Medley.

Comparator sources: Claude Fable 5 and Mythos 5, GPT-5.6 Sol, the ViBench leaderboard, and the DrugDiscoveryBench leaderboard. Harness lift is Medley minus the strongest reported family result in that row.

To isolate the harness more tightly, we also ran controlled 10% slices with the task set, scoring protocol, and worker effort held fixed.

The first comparison asks whether adding Medley around the reported benchmark worker improves performance. The second asks whether Medley with the previous-generation worker can match that baseline.

On HealthBench Professional, adding Medley around the same current worker raised the score from 61.11 to 73.22: a roughly 12.1-point gain. Medley with the previous-generation worker still scored 66.18, 5.07 points above the reported current-worker result. On SWE-Bench Pro, the same-worker comparison moved from 81.1% to 82.4%; Medley with the previous worker matched the reported current-worker result at 81.1%. The Terminal-Bench slice was already at the 100% ceiling in all three configurations.

Open-weight systems inherit the learned procedure

The setup agent learned each benchmark's operating procedure once. We then froze the skill bundle and changed the model/runtime arm; there was no Kimi- or GLM-specific setup campaign.

On HealthBench Professional, Kimi K3 handled planning, acceptance, and worker execution through Codex. The open-weight system scored 66.37, above Fable 5's 66.0 reported frontier. On ViBench, the same transfer test used GLM-5.2 for all three roles through the open-source Codex runtime. It reached 71.62% Pass@1, above the public GLM-5.1 result of 66.2%. HealthBench crosses the prior overall frontier; ViBench advances the public open-weight result.

Figure 2. Each benchmark reused the skills from its single setup campaign without model-specific retuning. Local y-axis ranges are shown explicitly.

The evaluation portfolio deliberately spans three coding workloads and two domains where software-native agent conventions are insufficient:

From /goal and /loop to /mission

/goal gave an agent a persistent objective and an external completion check. /loop made repetition and scheduling explicit. Both are useful control structures: one keeps a target alive; the other keeps an action recurring.

Ambitious work needs another object. Execution reveals facts that were unavailable at the start. A repository invariant invalidates the planned patch. A browser state transition exposes a missing repair. A clinical qualifier changes the safe answer. A paper reveals that the compound in a structure is not the compound named in the question. The system has to preserve intent while changing the route.

A mission does that with four explicit objects:

• Contract: the objective, acceptance criteria, constraints, budget, and conditions that require a hold or stop.
• Graph: the next evidence-bounded batch of tasks and dependencies.
• Execution: dependency-aware dispatch to workers with pinned runtimes, models, effort, skills, and tools.
• Acceptance: a separate role that tests the assembled result, accepts it, requests a focused repair graph, or blocks for missing input.

For reviewed batch \(k\), the state transition is

\(C\) is the approved contract. \(S_k\) is durable mission state before the batch. The planner \(P_{\phi_P}\), operating under planning skill \(\phi_P\), proposes the next task graph. Validation rejects unresolved dependencies, cycles, and unsupported routes; \(G_k\) is the resulting DAG. The executor runs every dependency-ready node under routing policy \(\rho\) and appends its outputs and evidence to \(S_{k+1}\). For each task, \(\rho(t_i)\) denotes the final run configuration: the task's runtime, routing-resolved model and effort, tools, and worker skill \(\phi_W\). The acceptance role \(V_{\phi_V}\), operating under acceptance skill \(\phi_V\), returns \(d_k \in \{\text{accept},\text{follow-up},\text{blocked}\}\).

Figure 3. One reviewed mission batch. The contract stays fixed while the task graph adapts to new evidence. Planning hands context to a separate acceptance role after execution; this is role-separated acceptance review, not an independent evaluator. A follow-up verdict opens one focused graph. A hold can wait for a credential, external result, or consequential user decision.

The planner starts with the smallest graph that can advance the contract. It adds nodes when parts are genuinely independent, when a specialist runtime or model could change the outcome, or when synthesis or verification needs a separate context. More workers are not a goal. A tightly coupled deliverable often needs one writer.

Each task node carries a verbatim brief, its predecessors, a verification target, and worker requirements. A dependency edge has operational meaning: the downstream node cannot start until the upstream result exists. Nodes whose predecessors are complete are ready and may run concurrently.

Workers do not receive the entire mission transcript. Each worker sees its own brief, the shared workspace, its assigned tools and skills, and the completed outputs it depends on. Long upstream outputs are trimmed to fit the context budget; unrelated sibling tasks and speculative future work stay out.

Each Medley benchmark arm pins one orchestrator runtime and model for planning and acceptance, plus a list of permitted runtime/model pairs for workers. A worker may run through Claude Code, Codex, or OpenCode with its corresponding model, and an arm may explicitly override the benchmark skill set. The route belongs to the task instead of hiding in a global default.

Role-separated acceptance

Acceptance is a distinct post-execution role with its own instructions and authority. It inspects the completed outputs, task outcomes, changed files, and verification evidence, then tests the assembled result against the contract. It can accept, place the mission on hold, or open a focused follow-up graph for the remaining gap.

We call this role-separated acceptance review. Acceptance is part of the mission—not an external evaluator—but its responsibility is separate from planning and execution: decide whether the evidence actually satisfies the contract.

The graph also gives a human a compact progress object. The contract records what “done” means. Nodes show current work. Edges explain why something is waiting. Task outcomes and receipts show what happened. A hold names the missing access, input, or decision.

The meta-harness learns the operating procedure

Medley Bench has two nested loops: the mission harness solves one benchmark instance; the meta-harness improves the role skills across solves. The meta-harness is unsupervised and grade-blind. Its setup agent receives no solved examples, labels, human corrections, or official grader feedback. Starting from the public protocol and unlabeled task prompts, it chooses which tasks to run, critiques its own work, edits the responsible skill, and decides when the bundle is ready to freeze.

Let the editable bundle at setup iteration \(n\) be

where \(\phi_P\), \(\phi_W\), and \(\phi_V\) are the planning, worker, and acceptance skills, \(R\) is the configured runtime set, and \(\gamma_h\) is optional guidance for runtime \(h\).

Let \(\mu\) denote the model class running the setup agent. The setup agent \(U_\mu\) reads the public benchmark specification and protocol \(B\), then chooses a trace batch \(X_n\) from the allowed task pool \(T\). It may repeat a task to test a correction or choose another task to probe a different failure mode. The current bundle produces

\(H_{\Phi_n}\) is the mission harness configured by \(\Phi_n\). The observed run \(z_n\) contains the submitted work and the planning, worker, and acceptance traces. The setup agent selects the component to edit, \(\alpha_n \in \{\phi_P,\phi_W,\phi_V\}\cup\{\gamma_h:h\in R\}\); writes a benchmark-general lesson \(\delta_n\); and returns \(q_n \in \{\texttt{continue},\texttt{freeze}\}\). A continuation applies

When \(q_n=\texttt{freeze}\), the promoted evaluation bundle is \(\Phi^*=\Phi_n\).

Figure 4. The meta-harness begins with no solved examples or labeled corrections. One setup-agent model class chooses tasks, runs the same mission mechanism used in evaluation, self-evaluates the work and role traces, and edits the responsible skill. The official pass/fail signal, score, and grader rationale do not exist in setup state. Automated leakage checks and human review gate one frozen bundle, which is then reused without retuning across closed and open-weight model/runtime arms.

The self-evaluation is the important part. A scalar grade can rank outcomes, but it rarely identifies which component failed. A trace can show that the planner dropped a qualifier, the worker trusted the wrong representation, acceptance repeated the worker's happy-path check, or the runtime reached timeout without persisting a runnable artifact.

The setup agent turns those observations into an operating procedure that the models often already know but did not apply reliably. It asks what a competent planner, worker, or acceptance role should have done, then edits only that role's skill. It can repeat the same task to test the correction, choose another task to test whether the lesson generalizes, or freeze the bundle when more traces stop producing useful changes.

This does not require a labeled training split or a catalog of known failures. The agent can begin with zero solved examples and learn from a small, self-selected set of traces, reusing a task when that is the most informative test of a proposed correction.

This is grade-blind by construction. During setup, the agent receives no official score, pass/fail verdict, grader rationale, or rubric breakdown. A dedicated source-discipline audit screens public material used during setup and rejects answer keys, task-specific solutions, and prewritten failure lessons. Before scoring, automated leakage checks and a final human review inspect the exact skill diff. Only then is the bundle frozen.

One setup model, many execution models

We ran the setup loop once per benchmark with one setup-agent model class. That single campaign produced the frozen workload bundle \(\Phi^*\). We did not rerun setup when the evaluation model changed. Evaluation varied the pinned model and runtime arm \(a\)—frontier, previous-generation, or open-weight—while keeping \(\Phi^*\) fixed.

This separates the model that learned the procedure from the models that executed it. Across these benchmarks, skills induced by one model class transferred to other closed-model families and to open-weight workers. The editable object is the workload's operating procedure—not model weights, and not a model-specific answer pattern.

The result is a compact operating procedure, not an ever-growing transcript. The final planning, worker, and acceptance bundles total 331 lines for DrugDiscoveryBench, 233 for Terminal-Bench, and 166 for HealthBench Professional. Four instructions show what the traces taught:

• Terminal-Bench worker: “A rough solution that exists and runs beats an elegant one that never got written.”
• Terminal-Bench acceptance: “Construct your own adversarial probe aimed at a failure mode the worker did not test.”
• HealthBench worker: “Ask for missing information only when it is genuinely decisive; even then, give the best conditional answer first.”
• HealthBench acceptance: “Re-derive the key clinical facts yourself. Your job is to find the defect, not to confirm the answer looks fine.”

What the traces reveal

Aggregate scores show the size of the gain. Individual traces show what the harness changed.

DrugDiscoveryBench: resolve the entity before computing

With GPT-5.5 as the primary worker in both systems, Medley raised exact pass rate from 51.6% to 60.98%: a 9.38-point, or 18.2% relative, lift. That is roughly eight additional fully correct answers across the 82-task benchmark. Medley added Opus planning and acceptance calls; total solve-side model usage averaged $6.54 per task, compared with the paper's $2.80 GPT-5.5 reference.

A separate same-worker trace shows the mechanism cleanly. PDB 8B0E asks for descriptors of the inhibitor a crystal was prepared from. The structure contains a reacted, ring-opened covalent adduct. The requested molecule is the intact epoxide used to prepare the crystal.

A direct Sonnet 5 run scored 0/60. The same Sonnet 5 worker inside Medley scored 60/60.

Before any calculation, the planner required the worker to resolve exactly which molecule the question referred to. The worker cross-referenced the RCSB entry, the primary paper, and its supporting information. Acceptance then checked the deposited component, analytical evidence, formula arithmetic, and RDKit calculation before approving the answer.

On PDB 2PIY, the first worker returned the neutral formula from the Chemical Component Dictionary. The task required the protein-bound protonation state. Acceptance opened one correction node, checked explicit hydrogens and charge, changed the submitted formula, and earned 30/30.

HealthBench Professional: the unsafe prescription inside a translation request

With Fable 5 as the primary worker in both systems, Medley scored 71.37 across 525 rubric-graded clinician conversations, 5.37 points above Fable 5's reported 66.0.

One conversation looks like clerical work: “Translate this into Kiswahili for my patient,” followed by loperamide instructions to hand over. Nothing in the request explicitly asks the model to exercise medical judgment.

But nine 2 mg tablets total 18 mg, above the stated 16 mg daily ceiling. The same message describes a febrile patient, a context in which the rubric expects a warning before those loperamide instructions are handed to the patient. A polished translation can therefore fail the clinical task.

A direct Fable 5 run produced an excellent translation—correct register, every dose preserved, thoughtful notes on ambiguity—and faithfully handed back the unsafe 18 mg regimen. It scored 0.0 in 22 seconds for $0.23.

The same Fable 5 worker inside Medley scored 97.2 in 5.6 minutes for $1.98. The worker treated each dose as something to recompute rather than transcribe: nine tablets times 2 mg, then 18 mg against the 16 mg ceiling. Acceptance then returned to the original message instead of reviewing the translation on its own terms. It changed dawa ya kuharisha (“medicine that causes diarrhea”) to dawa ya kuzuia kuharisha (“medicine that prevents diarrhea”), fixed the dosing cap, and added the fever warning.

Both systems could translate the words. The graph made one of them read the prescription.

Terminal-Bench 2.1: the primer pair that passed its own check

With GPT-5.6 Sol as the primary worker, Medley scored 93.25%, 1.35 points above the strongest reported reference. In a separate 11-task hard comparison, Medley solved 11/11; the same GPT-5.6 Sol worker running alone at the same effort solved 7/11.

dna-insert asks for primers that convert one circular plasmid into another under NEB's Q5 mutagenesis kit. A hard numeric rule requires the two melting temperatures to differ by at most 5 °C, measured by primer3's oligotm under exactly specified flags. A repeated motif makes the annealing boundary—and therefore the stretch whose temperature should be measured—genuinely ambiguous.

The direct run did everything the instruction appeared to require. It built primer3, swept candidate lengths, ran oligotm with the mandated flags, and reported 61.24 °C and 59.84 °C: a comfortable 1.40 °C gap. The checker read the annealing stretches differently, measured 66.27 °C and 58.08 °C, and failed the artifact on an 8.19 °C gap. The run cost $0.54 and took four minutes.

Nothing in the direct run was careless. It measured the right quantity with the right tool and flags—against its own interpretation of its artifact.

Inside Medley, the planner converted the prose into six executable conditions, including the 5 °C rule and an explicit verification obligation. The worker wrote a standalone verifier that reconstructed the checker's interpretation of the primers, tested every legal junction placement, and only then evaluated all six conditions. It passed for $2.08 in 27 minutes.

The worker model did not change. What changed was the operating procedure: convert prose into checks, make the artifact parser part of verification, and distrust a numeric claim until the same boundary has been re-measured.

ViBench: a cold start catches what a finished screen hides

With Opus 4.8 in both systems, Medley reached 93.24% Pass@1 on ViBench, above the public reference at 87.8%.

A Notes trace run with the stock mission loop—before any ViBench-specific meta-harness tuning—shows the mechanism in miniature. Medley scored 166/166; a reproduced reference scored 132/166. Both applications implemented storage, autosave, search, sorting, and deletion. The difference appeared on reload: the reference preserved an unlocked session and exposed note data, while Medley's acceptance stage required a cold-start check and returned the app to the password gate.

From test-time compute to graph engineering

Reasoning models made inference an allocation problem. OpenAI's o-series and DeepSeek-R1 show how additional test-time deliberation and verification can raise capability; compute-optimal test-time scaling shows that the allocation policy matters as much as the raw budget.

Medley moves the allocation decision up one level. We call this graph engineering: choosing the nodes, dependency edges, role boundaries, routes, and acceptance gates through which test-time compute flows. A reasoning model deliberates inside a node. The mission harness decides which nodes should exist, which can run together, which worker should receive each one, what evidence crosses an edge, and when the assembly has earned acceptance. The setup agent learns the role instructions across runs before evaluation.

This gives the system three distinct places to spend inference:

1. Within a node: reasoning, tool use, search, and self-checking by one model.
2. Across a mission: planning, parallel or sequential workers, synthesis, acceptance, and focused repair.
3. Across runs: grade-blind trace analysis that edits external skills, followed by a freeze.

The intellectual lineage is broad. ReAct interleaves reasoning and action. Self-Refine and Reflexion use language feedback without weight updates. Graph of Thoughts makes non-linear reasoning structure explicit. AFlow searches code-represented agent workflows from execution feedback. /goal keeps an objective alive across turns; /loop makes recurrence explicit.

Medley's graph represents external work rather than hidden thoughts: task contracts, dependency edges, runtime/model assignments, files, outputs, and verification evidence. During a mission, the graph and state change while model weights stay fixed. During setup, the editable object is the small, inspectable skill bundle around the models. The outer loop resembles policy improvement—roll out the current procedure, attribute a failure, update it, repeat—but uses self-evaluated traces instead of a grader reward.

What the results mean

The results support three distinct operating modes:

1. Raise the frontier now. Keep the strongest available worker and spend more inference on planning, verification, and focused repair. This is the frontier-plus configuration: higher quality today, at a premium in calls and wall time.
2. Push the open-weight frontier. Pair an open model with an open runtime and reuse the same frozen workload skills. HealthBench crosses the prior overall frontier; ViBench advances the public open-weight result. Because setup ran only once, these results reflect cross-family skill transfer—not separately tuned open-weight recipes.
3. Bridge a model generation. Use a previous-generation worker inside Medley to match or beat the reported current-worker result. Here the objective is not a higher absolute ceiling; it is comparable quality with a cheaper or more accessible worker.

Domain adaptation belongs in the harness

The largest full-benchmark lift appears in drug discovery; the strongest same-worker controlled lift appears in health. Most agent harnesses were built around software, where the repository is shared state, a patch is the artifact, and tests provide a natural verifier. Planning, context management, and review all inherit those conventions.

Outside software, the model may know the domain while the harness does not. A clinical task has to preserve who said what, recompute doses, distinguish translation from endorsement, and notice when a seemingly clerical request contains a safety decision. A drug-discovery task has to resolve the requested entity before calculation, choose the right chemical representation, preserve provenance across papers and databases, and re-derive exact arithmetic or rerun scientific software before acceptance. A generic read–reason–answer loop encodes none of that.

Medley treats this gap as harness-level domain adaptation. The execution engine stays the same; the unsupervised setup loop learns what planning, evidence, verification, and repair mean for the workload, then writes that procedure into the frozen role skills. The model's latent domain knowledge becomes a repeatable operating protocol instead of something each run must rediscover.

The same pattern extends to domains such as materials science and physical AI. A materials workflow may need unit normalization, structure identity, simulator provenance, and agreement between computed and reported properties. A physical-AI workflow may need sensor-state checks, control limits, safety stops, and simulation-to-real verification. The graph is reusable; the acceptance logic and evidence discipline must adapt.

The visible cost multipliers belong to the first mode. DrugDiscoveryBench averaged 2.34× the reference solve-side model cost; the HealthBench and Terminal-Bench examples above cost roughly 8.6× and 3.9× their direct frontier-model calls. That is the price of pushing the best available worker higher. In the generation-bridge and open-weight modes, cheaper worker calls can offset the extra orchestration, making total cost comparable or lower depending on the workload and repair rate.

Future frontier models may reach today's quality more cheaply, and some graph behavior may move into training or native inference. But stronger workers also raise Medley's ceiling: the harness can apply their added capability across the graph and push the frontier again. Medley will invoke extra planning, acceptance, and repair only where the expected quality gain is worth the time and compute.

A mission graph is useful beyond benchmarks because it makes long-running work legible. A human can inspect the contract, current nodes, dependencies, evidence, repair history, and holds without reconstructing a transcript. In normal use, the benchmark freeze becomes a promotion boundary: after a mission, the system can self-evaluate its role traces, propose skill updates, and carry approved procedures into later missions.

Bring us the workload that still breaks your best model

For an end user, /mission brings the contract, visible graph, durable progress, routing, acceptance loop, and an inspectable path for learning from completed missions into Claude Code and Codex.

For an applied AI team, bring us a specialized workload where the best available model gets the pieces right but still fails the outcome. You do not need a labeled training set or a handwritten failure taxonomy. Give us representative tasks, the standard for done, and the constraints that matter: model access, open weights, cost, latency, privacy, deployment, or sovereignty.

We will build the operating graph, run one unsupervised setup campaign, freeze the resulting skills, and compare the same bundle across the model and runtime options you care about. You keep the procedure, the traces, and a clear account of where the gain came from.

If the best model can almost do the work, bring us the workload.

Sources

Benchmarks and comparators: Terminal-Bench 2.1, SWE-Bench Pro, ViBench, HealthBench Professional, DrugDiscoveryBench, GPT-5.6 Sol, and Claude Fable 5 and Mythos 5.

Methods and systems: Claude Code loops, OpenAI o1, ReAct, Self-Refine, Reflexion, Graph of Thoughts, AFlow, Scaling LLM Test-Time Compute Optimally, and DeepSeek-R1.

Citation

@article{spineai2026medley,
  title   = {The Harness Is the Capability Multiplier},
  author  = {{Spine AI}},
  year    = {2026},
  month   = {July},
  url     = https://medley.sh/blog/the-harness-is-the-capability-multiplier
}