Working Surfacelink · 12 Nov 2026
02
A tireless clockwork automaton rebuilds a tall tower of wooden blocks over and over, while above it a single framed table of expected measurements hangs locked inside a glass case whose key a figure seen from behind holds, the whole scene drawn with wide empty margins on every side.
martinfowler.com

Tests as data: Ensuring AI-generated code does what we want

Matteo Vaccari · 11 November 2026

How-to · critique & review · artifacts · Ground truth for review

Read the original
Takeaway

Matteo Vaccari of Thoughtworks proposes that developers write plain-language customer tests, kept as protected data files, as the contract that AI coding agents must meet.

Summary

Matteo Vaccari, a developer and Technical Principal at the consultancy Thoughtworks Italia, interviewed 26 developers at an organisation that relied heavily on AI coding agents. They told him they did not trust their tests, because the agents changed the tests whenever they changed the code. On martinfowler.com, Vaccari proposes keeping the important tests as readable data files that the agents cannot quietly edit.

Key points
  • Vaccari cites 2026 industry reports of rising defects, including a 28.7% rise in bugs per pull request reported by Faros AI.
  • He argues that AI-written tests are long, hard to read and tied to the current code, so they often confirm existing bugs.
  • His customer tests state inputs and expected results in terms a domain expert can read, often as tables, and an AI agent writes the program that runs them.
  • Teams can make the test files read-only to the agent, or run a check in continuous integration that flags unexpected changes to them.
Implication

A team that lets AI agents write most of its code can keep the human-readable statement of correct behaviour in files that people own and the agents cannot change. Vaccari notes that this is one practitioner's view and that prototypes can skip it.

Plays
Engineering

Write customer tests as tables of inputs and expected results in plain terms, and have an AI agent write the program that runs them.

Engineering

Make test data files read-only to coding agents, or add a continuous integration check that flags unexpected changes to them.

Derived by Working Surface from the article; more in the Playbook. Source line: When AI writes the code, the developers become in a way customers.

Source issue

12 November 2026: when AI agents write the code, engineers' main work becomes writing and protecting the checks that decide whether that code is correct.