Is the Agent Harness Really Dead? Three Sets of Experimental Data and the Data Flywheel
Is the Agent Harness dead? This was the fiercest debate in the programming world in the second half of 2026, and the beginning of a lot of poor tooling decisions: some people used it as a reason to strip every guardrail from Claude Code, while others simply switched to an ultra-minimal tool that could only read files, write files, edit files, and run commands.
This article breaks the answer down into something actionable: first, three sets of public experiments show that harnesses have almost no effect on scores but a major effect on the bill; then we explain the data flywheel behind it, and finally identify three things models will never absorb and two actions you can take today. The conclusion may run counter to intuition—what should become thinner is not how you work, but the flashy instruction manual wrapped around the model.
I. What Exactly Is an Agent Harness? The Layer Around the Model
Literally, a harness is horse tack—the reins and shafts fitted to a horse. A horse may have the strength, but without that equipment, it cannot pull a cart. In AI, the model is the horse, while the Harness is the layer of software wrapped around it that lets it actually get work done.
It handles all sorts of things: which prompts to provide, which tools are available, how memory is stored, how to compress an overly long conversation, what to block from being done, which machine runs the task, and how to resume after an interruption. Claude Code, Codex, and Cursor Agent, which you use every day, are essentially a model plus a Harness. The model can change, but this outer layer determines how smoothly it works in practice.
A more practical way to break it down is into two layers:
| Layer | What It Handles | Nature |
|---|---|---|
| Runtime layer (runtime) | Where execution happens, how state is stored, how to recover after an interruption, sandboxing, and process lifecycle | Traditional software engineering; already mature |
| Context and orchestration layer (context / orchestration) | Which model to call, what to show it, tool definitions, memory compression, behavioral interventions, multi-Agent collaboration | The “Wild West”; no settled answer |
Almost the entire “Harness is dead” debate concerns the second layer. Nobody has denied the first.
II. Three Names in Three Years, but the Same Job
The hottest job title in 2023 was prompt engineer. By mid-2025, it had become context engineer—someone responsible for assembling the most appropriate material before every call and feeding it to the model. Only in early 2026 did the term Harness engineer emerge.
Every rename represented an expansion in scope: writing a good instruction → selecting what the model should see → designing the model’s entire execution environment. The name changed, but the essence did not: how best to build the layer around the model.
Garry Tan, president of YC, offered a pointed take on this direction:
If your memory dies along with the Harness, your Harness is too thick. Memory is Markdown, and skills are Markdown too. The Harness is merely the conductor that reads those files; it does not own them.
His formulation was “thin harness, fat skills”—a thin Harness and fat skills. The phrase would eventually become something like a slogan for the entire debate.
III. The First Evidence for “The Harness Is Dead”: The People Who Understand It Best Are Cutting Back
On January 14, 2026, Cursor published Scaling long-running autonomous coding. They used a long-running harness, internally called Grind Swarm, to orchestrate large numbers of Agents and machines as they built a browser from scratch: it ran for a week, produced 1,000 files and more than one million lines of code, and consumed a number of tokens measured in the trillions.
This was the approach in early 2026—when the model was not strong enough, compensate with the Harness. Hundreds of Agents divided up the work and coordinated with one another, working miracles through brute force. The cost was spelled out in Cursor’s own blog post: the demo was extremely expensive and overwhelmed Cursor’s own infrastructure at least twice.
Then the tide turned.
On April 30, 2026, Cursor published an article about how it continuously improves its Agent Harness (by Stefan Heule and Jediah Katz). It was unusually candid: in the early days, because models could not select context well, Cursor had piled on guardrails—forcing lint results and type errors back into the context after every edit, rewriting its file-reading behavior whenever the model read too little, and even capping the number of tools that could be called in a single turn. The article’s summary was blunt: “That is mostly long gone.” (Most of those are now gone.)
In late July 2026, Thariq from the Claude Code team posted a message: to accommodate a new generation of models, they had deleted roughly 80% of Claude Code’s system prompt, and the new models still performed just as well.
The conclusion is clear: “The Harness is dead” is not empty rhetoric; the companies that understand Harnesses best are all cutting back.
IV. The Second Set of Evidence: The HarnessTax Test—21 Combinations, Almost No Change in Scores, but Twice the Cost
But “everyone is cutting back” is only motivation; we need data. On September 16, 2026, the teams from UC Berkeley and Arena released the HarnessTax study, asking a simple question: if you swap the Harness around the same model, can it solve more problems correctly or spend less money?
The experimental design was tightly controlled: 7 models × 3 Harnesses = 21 combinations, with the evaluation stack fixed and only the Harness changing. The researchers ran two public coding benchmarks: SWE-bench Lite and Terminal-Bench 2.0. The three Harnesses were Claude Code, OpenAI’s Codex CLI, and the open-source minimal tool Pi—whose entire toolkit consisted of four tools: read files, write files, edit files, and run commands.
There were three conclusions:
First, switching Harnesses had little effect on the number of tasks solved. The average difference was within 2 percentage points on SWE-bench Lite and within 5 percentage points on Terminal-Bench 2.0.
Second, it had a major effect on cost. For the same model, Claude Code cost about 2.0× as much as Pi and 1.6× as much as Codex on SWE-bench Lite. On Terminal-Bench 2.0, it cost about 1.5× as much as Pi.
Third, the most striking data point came from the same strongest Claude model:
| Harness | Success Rate | Cost per Attempt |
|---|---|---|
| Claude Code | 97.8% | $1.33 |
| Codex CLI | 96.7% | — |
| Pi | 96.7% | $0.67 |
The difference was 1.1 percentage points—roughly one additional correct answer for every 90 questions—but it cost twice as much.
Where does the extra cost come from? The mechanism is right there in the data: the content Claude Code passes into the first model call averages more than 10× what Pi passes in—a longer instruction manual and more tool definitions. Before the work even begins, it has read a manual ten times longer.
Portkey broke the bill down in even greater detail in an April post: on the same task, Pi used about 2,600 input tokens, Codex about 15,000, and Claude Code about 27,000. In a 40-turn long-running session, Claude Code burned through about 1.12 million input tokens, nearly half of which were consumed by the harness itself, unrelated to your task.
This extra overhead has a name: harness tax—the portion of an Agent’s cost spent on “itself” rather than on your task.
V. The Third Set of Evidence: EPFL’s MLE Experiment—A Carefully Built Machine Cannot Outperform a Minimal Agent
The second study came from the Swiss Federal Institute of Technology in Lausanne (EPFL) and was posted to arXiv on September 30, 2026: How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering? (Kirill Brilliantov, Alejandro Hernández-Cano, and Emmanuel Abbé).
It tested whether AI could perform machine-learning engineering (MLE) on its own. Agents in this field often sit atop a complex machine: a multi-Agent orchestrator, dedicated retrieval sub-Agents, and a pile of rules and guardrails.
The paper’s conclusion was that under the same time budget and the same frontier-model backbone, an open-source state-of-the-art Harness had no advantage over “a single-session minimal coding agent that could only read, write, and run bash”—the bottleneck was the backbone model, not the machinery layer. A series of large-scale ablation experiments argued that the machinery layer was redundant in the coding-agent setting.
Two teams, two completely different fields, the same conclusion. In both the coding and machine-learning engineering lanes, the flywheel has completed a full turn.
That said, the other side of the ledger needs to be spelled out. One study found that on long-running tasks, changing only the Harness while keeping the model fixed could raise Terminal-Bench 2’s pass@1 from 69.7% to 77.0%. Another benchmark reported that scaffolding alone could produce swings of 15 percentage points. These findings do not conflict; they differ in what they are comparing: the convergent finding compares “several mainstream CLIs,” while the divergent finding compares “a well-built Harness” with “a bare, badly configured Harness.” Cursor’s own summary is blunter: once those behaviors became native model capabilities, the patches stopped earning their complexity.
VI. The Data Flywheel: Why the “Coding” Lane Has Completed One Turn
All of those reductions have the same mechanism behind them, which the industry calls a data flywheel:
- The Harness unlocks capabilities first. For things the model cannot do on its own, prompts, tools, and guardrails give it enough support to take on harder work.
- The Agent takes on new tasks. Once it can perform them, more people start using it.
- Those tasks become training data. The code Agents write every day flows into the labs that build models.
- The model absorbs the capabilities. How to find what it needs in an unfamiliar codebase, how to work for hours without going off track, how to run its own tests after making changes—once these had to be taught line by line in the Harness, but now the model knows how to do them.
- The Harness removes the scaffolding it originally added. The flywheel then returns to step one, finds the next thing the model still cannot do, and supports it there all over again.
So “coding Harnesses are getting thinner” is not a failure; their job is done. One of Cursor’s leads put it this way: as models became smarter, the Coding Agent Harness went through a period of stagnation, but now the flywheel is back at the starting point.
By the way, this mechanism explains a counterintuitive phenomenon: product requirements often run ahead of model capabilities, so the Harness becomes fat before it becomes thin—once the model catches up, the scaffolding should retire.
VII. The Reverse of the Same Logic: Why Grok Bot Is Adding Guardrails Back
If Harnesses only ever became thinner, every product built on top of them should have become simpler. Reality is the opposite.
On August 11, 2026, SpaceXAI released an early beta of Grok Bot: an AI teammate with a computer of its own that signs into your various tools, uses them the way you do, and then hands the completed work back to you.
It does not write code; it handles the kinds of work you do in an office: making presentations, booking flights, and emailing colleagues.
Precisely because the model has not mastered these new domains yet, what the Grok Bot team did was exactly the opposite of what Cursor did back then—they added prompts and guardrails back one by one:
- How web workflows proceed: Tasks such as making presentations or booking flights require reliable interaction with external systems.
- How often to report progress: Keep you informed about what it is doing without interrupting you constantly.
- When to act independently and when to pause and ask: You want an Agent to email a colleague, but only in the way you intended.
- Memory: The system must gradually learn how you work.
The architecture is different too: an always-on coordinator directs a group of worker Agents, rather than having one chat window per request. Think of it like hiring a highly capable lawyer who interfaces with you directly, sets the overall direction, and delegates specific work to a team of assistants.
The value of a Harness is therefore domain-specific: wherever the flywheel has reached, the Harness becomes thinner; wherever the model has not yet learned, the Harness grows thicker. The same logic, pointing in two directions.
VIII. Three Things Models Cannot Absorb
In new domains such as Grok Bot, the Harness still has to grow. But there are three things that no model, however strong, can absorb. They inherently belong to the layer around the model:
First, where it runs. An Agent task that used to take minutes can now run for hours, days, or even longer. Work like that cannot live on your laptop: close the lid, and the computer goes to sleep, interrupting the Agent. This requires cloud machines, persistent state, and the ability to resume after an interruption. This is infrastructure, not intelligence. No amount of intelligence can keep a powered-off process alive or conjure up state that was never stored. SpaceXAI puts it even more directly: within six months, 90% of Agent work will run beyond the laptop.
Second, what it can access. The earlier minimal Harness—one shell with every secret in it—does well on coding tasks. But you probably would not put every secret in a single file and hand it to a model. Even if you do not mind, your company’s security team certainly will. Nobody wants to wake up one morning and discover that an Agent did something irreversible. Someone therefore has to draw the boundaries around what each Agent and each person can do, and this can only be done in the Harness. Prompts can influence behavior, but only the Harness can control the credentials and actions that are actually available.
Third, what it costs. The scores converged; the costs did not. The Grok Bot team spent the past month continuously optimizing cache hit rates, token consumption, and model selection, bringing down both costs and latency without reducing quality. The more work you give Agents, the more valuable this layer becomes. A small difference may not matter when an Agent runs only occasionally. Once it takes over much of your daily work, it becomes a genuine product and infrastructure issue.
None of these three depends on how intelligent the model is. Scores can converge; runtime, permissions, and bills cannot.
IX. What Future Harnesses Will Look Like
The direction is clear, with roughly two paths.
Direction one: from “composing model calls” to “composing Agents.” In the past, a Harness orchestrated a collection of model calls. In the future, the smallest orchestrated unit will be an Agent, with each sub-Agent having its own context, tools, and execution loop, while the upper layer handles task assignment and lifecycle management. An internal Cursor example illustrates this architecture particularly well:
Someone reports a bug in a group chat → it is automatically triaged → a cloud Agent is spun up to fix it → once fixed, it records a before-and-after comparison video as evidence → after the code is merged and deployed, another Agent is spun up to watch production for problems.
No single model call can own the entire lifecycle. This kind of “Harness made of Harnesses” is called a meta-harness.
Direction two: increasingly complex on the inside, increasingly invisible to the user. Every exposed detail becomes something people will depend on (Hyrum’s Law), making it impossible to change later. The Harness therefore needs to learn how to hide its internal mechanisms while providing composable primitives, such as a unified “subscription primitive”: when an Agent needs to be awakened because someone sends a Slack message, a PR’s CI fails, or a certain time arrives, it should not need a separate integration for every event source.
A “retirement mechanism” will support this process as well: temporary behavioral patches, known as shims, are cataloged centrally by the team, with every trigger logged. A Bot then watches the dashboard and automatically opens a PR proposing the removal of a shim once it no longer fires. The retirement of scaffolding is automated too, allowing code to be deleted as soon as the model improves instead of leaving behind a pile of fossilized guardrails.
X. Applying This to Your Own Work: Two Actions You Can Take Today
There is no point in leaving all of this at the conceptual level. Two small things can actually change your bill and your experience.
First, write the rules, requirements, and pitfalls you encounter with AI as ordinary files in your own project whenever possible, rather than storing them only in one tool’s settings. That way, when you switch Harnesses tomorrow, they travel with you—this is exactly Garry Tan’s “fat skills, thin Harness.” In my own post, Pi vs. Oh My Pi: Choosing Between a Minimal Core and an Out-of-the-Box Fork, the one criterion I insist on carrying from one tool to another is also this: whether your rule files can be moved over unchanged.
Second, take a look at your bill. If you pay per use, pick a task you do regularly, run the same model once in each of two tools, and compare the number of tokens used. If you pay for a subscription, see how quickly it consumes your allowance. The difference may be much larger than you expect—the data above showed one side costing twice as much for the same score. Running your own test will be more accurate than any leaderboard because the public numbers measure other people’s tasks. If you want to see what a real-world bill looks like, see MiniMax M3.1 Flash Burned 7 Billion Tokens in Three Weeks: A Practical Test.
The same applies at the tool layer: choose portable formats. Plain-text rules, Markdown skill files, and MCP tools such as REA that use open protocols are the things you can take with you when switching Harnesses.
Summary
“The Agent Harness Is Dead” is an oversimplification.
In coding, it really is retiring. Across HarnessTax’s 21 comparisons, the number of tasks solved barely moved while costs could differ by a factor of two. EPFL’s MLE experiment likewise found no advantage for complex orchestration. The reason is not that Harnesses have become useless, but that the capabilities they once supported have been burned into the model’s weights by the data flywheel. The scaffolding has fulfilled its historical mission, so it should naturally be dismantled.
But there are three things it will never absorb: where it runs, what it can access, and what it costs. These are infrastructure, permission boundaries, and operating economics—not problems of intelligence.
And in new domains the model has yet to master, Harnesses are still adding guardrails back. What Grok Bot is doing is exactly the opposite of what Cursor did back then. This is not a contradiction; it is the same logic pointing in two directions.
For you, the conclusion can be compressed into one sentence: tools can change and Harnesses can become thin, but your own rules, memory, and skill files are assets. Put them into your own project as ordinary files, then spend an afternoon testing exactly which layer you are paying for.
The data in this article comes from three sets of public studies and first-party vendor materials: HarnessTax (UC Berkeley and Arena, 2026-09-16), the EPFL MLE Harness paper (arXiv:2609.40303, 2026-09-30), and official blog posts and announcements from Cursor and SpaceXAI. We also drew on AQ’s overview of harness tax and a written transcript of the Agent Harness talk. All percentages and cost figures come from those sources; conclusions may vary across tasks and models, so test with your own real-world tasks before making a purchasing decision.
If you have also run into the problem of “the bill doubling when you switch tools for the same task,” feel free to share your comparison in the comments. If this article was useful, bookmark it or start with our comparisons of Pi versus Oh My Pi and MiniMax M3.1 Flash token usage.