Tests as data: Ensuring AI-generated code does what we want
Matteo Vaccari · 11 November 2026
Read the originalMatteo Vaccari of Thoughtworks proposes that developers write plain-language customer tests, kept as protected data files, as the contract that AI coding agents must meet.
Matteo Vaccari, a developer and Technical Principal at the consultancy Thoughtworks Italia, interviewed 26 developers at an organisation that relied heavily on AI coding agents. They told him they did not trust their tests, because the agents changed the tests whenever they changed the code. On martinfowler.com, Vaccari proposes keeping the important tests as readable data files that the agents cannot quietly edit.
- Vaccari cites 2026 industry reports of rising defects, including a 28.7% rise in bugs per pull request reported by Faros AI.
- He argues that AI-written tests are long, hard to read and tied to the current code, so they often confirm existing bugs.
- His customer tests state inputs and expected results in terms a domain expert can read, often as tables, and an AI agent writes the program that runs them.
- Teams can make the test files read-only to the agent, or run a check in continuous integration that flags unexpected changes to them.
A team that lets AI agents write most of its code can keep the human-readable statement of correct behaviour in files that people own and the agents cannot change. Vaccari notes that this is one practitioner's view and that prototypes can skip it.
Write customer tests as tables of inputs and expected results in plain terms, and have an AI agent write the program that runs them.
Make test data files read-only to coding agents, or add a continuous integration check that flags unexpected changes to them.
Derived by Working Surface from the article; more in the Playbook. Source line: When AI writes the code, the developers become in a way customers.
12 November 2026: when AI agents write the code, engineers' main work becomes writing and protecting the checks that decide whether that code is correct.