For most of the last two years, every time a coding agent did something stupid in my projects, my first reaction was the same: the model isn't good enough yet. Wait for the next release.
I don't think that anymore. Almost every failure I have had with agents this year was not a model problem. It was a harness problem. And a harness problem is, at its core, a testing problem.
What a harness is
LangChain put it in one line in The Anatomy of an Agent Harness: Agent = Model + Harness. The model holds the intelligence. The harness is every piece of code, configuration and execution logic that is not the model: the system prompt, the tools and MCP servers it can call, the files it can see, the hooks that run before and after each step, the memory it keeps, and the permissions that decide what it may touch.
When you use Claude Code, Codex or Cursor, you are not "using a model". You are using a harness that somebody built around one, and then you extend it with your own: a CLAUDE.md, a set of MCP servers, a deny-list, a few hooks.
The evidence: same model, different results
The clearest data point so far came from LangChain in February. In Improving Deep Agents with harness engineering they kept the model fixed and changed only the harness around their coding agent. Its score on Terminal Bench 2.0 went from 52.8 to 66.5, which they describe as moving from the top 30 to the top 5.
What they changed reads like a QA engineer's checklist:
- A verify step before exit. Their most common failure was an agent that wrote a solution, re-read its own code, decided it looked fine and stopped without running anything. They added a middleware that intercepts the agent before it finishes and makes it check its work against the task spec.
- Loop detection. After too many edits to the same file, the agent gets a nudge to reconsider its approach instead of digging deeper.
- Context up front. A step at the start maps the working directory and the available tools, so the agent stops guessing about its environment.
- Time budgets. Warnings that push the agent towards verification before it runs out of time.
None of this is AI research. It is test discipline, enforced by code.
OpenAI made the same point from the other side. In Harness engineering: leveraging Codex in an agent-first world Ryan Lopopolo describes a team that shipped a product where agents wrote the code, the tests, the CI and the docs, summed up as "Humans steer. Agents execute." The humans' job became building the environment that made that safe.
Three harness failures from my own projects
This site is built with Claude Code, Payload CMS and an MCP server that lets the agent read and write content. Here are three things that went wrong, and why upgrading the model would not have fixed any of them.
1. The update that erased my CV
I asked an agent to change two fields on the CV page through the CMS's MCP server. It called the update tool with exactly those two fields, as asked. The tool treated the call as a full replacement, not a merge, and every other field on the page went empty on the live site.
The model did nothing wrong. It used the tool exactly as described. The harness was missing two things: a rule saying global updates must send every field, and a verify step that re-reads the record after writing. Both are now part of the setup, and it has not happened again.
2. The post that production refused
Long blog posts sent through the same MCP server were rejected with a 403, while short ones went through. The cause was not size in bytes but the number of JSON fields in the request: rich text spends several fields per run of text, and the host's firewall refuses requests above roughly a thousand.
No model would ever guess that. The fix is a step in the harness that counts fields before sending and trims the optional ones. This very post went through that check.
3. The rules that live in a file
My repo has a CLAUDE.md that reads like a list of scars: never run the production build while the dev server is running, keep a specific bundler flag or a font silently stops loading, never import the design system from the wrong file or every heading reverts to the wrong size. Each line exists because an agent (or I) hit that wall once.
That file is not documentation for humans. It is test fixtures for the agent's context. It turns a one-off failure into a rule the agent cannot miss.
Why this belongs to QA
Look at what the harness actually does: it defines the environment, constrains the inputs, checks the outputs, detects loops, and blocks actions that should never happen. That is what a good test setup does for a system under test.
So when an agent fails, I now debug it the way I would debug a flaky test:
- Reproduce it. Keep the full transcript, not just the final answer.
- Find the layer. Was it missing context, a misleading tool, a missing permission boundary, or no verification step?
- Fix the harness, not the prompt of the day. Add the rule, the hook, the check or the narrower tool.
- Add a regression case. The same task should now pass, every time.
Models will keep getting better, and some of today's harness tricks will be absorbed into them. But a model will never know that your CMS replaces instead of merging, or that your host blocks big requests. Somebody has to encode that, test it and keep it working. That somebody looks a lot like a QA engineer.
What was the last agent failure you blamed on the model, and would a harness change have caught it?
