SpecBench: A tool to evaluate AI-enabled software development workflows
Yixiong Jiang, Chen Chen and Yuhua Li · 13 November 2026
Read the originalThoughtworks engineers built SpecBench to measure AI coding workflows and found that adding specifications and planning raised correctness slightly while nearly doubling the time taken.
Yixiong Jiang, Chen Chen and Yuhua Li work at Thoughtworks, a technology consultancy. Teams add steps such as written specifications, planning and review to coding with AI agents and need evidence that the steps pay for themselves. The authors built SpecBench, a platform that runs the same tasks through different workflows and scores the results independently.
- SpecBench holds the task, AI model, agent software and conditions constant and changes only the workflow, so the comparison isolates the workflow.
- Across frontend, backend, full-stack and command-line tasks, a workflow with specification, planning and task splitting raised behavioural correctness from 92.25% to 95%.
- The same workflow raised mean time from 84 to 164 minutes and tokens from about 450,000 to 1.78 million.
- The authors note its limits: few test cases, many frontier AI models untested, and no tests yet of changes to existing codebases.
The authors suggest that a team state what a workflow step should improve and check whether the improvement appears before keeping it.
Before adopting an extra step in an AI coding workflow, run the same tasks with and without it and compare correctness against time and token cost.
Derived by Working Surface from the article; more in the Playbook. Source line: The purpose is to make workflow development measurable.
14 November 2026: as AI agents produce more code and tests than people can inspect, teams are writing down which work gets checked closely, by whom or by what, at what cost, and how they will know it worked.