Are Harnesses Worth the Cost?

Naresh Jain, founder of testing company Specmatic, found that tests running during the build made a cheap AI model as reliable as a frontier one.

Naresh Jain, founder of Specmatic, which makes API contract-testing tools, had coding agents build the same backend service ten times in each condition. Half the runs received a harness: machine-readable specifications, examples and automated tests that report failures back to the agent while it works. Every finished service then faced the same 178 independent acceptance tests, and the article appeared on martinfowler.com.

• OpenAI's frontier GPT-6 Sol passed in 10 of 10 runs with or without the harness, and the harness made its runs about seven times more expensive.
• The cheaper GPT-6 Luna passed in 3 of 10 runs alone and in 10 of 10 runs with the harness.
• A harnessed Luna run cost about $0.273, against about $0.406 for an unharnessed Sol run.
• The agent was told to keep the human-written specifications and tests unchanged and to fix its own code instead.

Jain recommends running the tests throughout development for cheaper AI models and once, as a final acceptance check, for frontier models. The result comes from one new service with an unusually detailed brief, and the author's company makes the testing tool he used.

Derived by Working Surface from the article; more in the Playbook. Source line: What changes with model capability is when and how often the sensors run, not whether an independent check is needed.

14 October 2026: as AI agents raise the volume of code changes, teams are moving routine checks into automated tests and installable rules, and keeping people for intent and contested decisions.