Attempting to reverse engineer code review

Cate Huston, an engineering leader, used Claude to mine 4,176 code review comments and found that review missed most problems and enforced rules the team never agreed.

Cate Huston is a former Google engineer who writes about engineering leadership. She asked whether human code review could become an automated check that every change passes before a person looks. With Claude, Anthropic's AI model, she studied twelve reviewers on an open source project over 20 months and turned their recurring comments into 57 checks. She then replayed those checks against about 500 recent changes.

• More than half of the recent changes would have failed at least one check, and 71% of about 700 problems were never raised by a person or a review bot.
• In 92% of changes exactly one person reviewed the code, and individual reviewers caught between none and 38% of the problems in the changes they reviewed.
• The missed problems did not lead to more later bug fixes than other new code: 20 of 345 traceable cases, against about 17 expected.
• Some rules were contested, such as whether to read a feature flag while the app runs or once at launch, and those need a team decision first.

Huston would automate the checks nobody disputes, and settle contested rules and who decides them before they become automated failures. She doubts that one reviewer per change still works once AI agents write the code.

Derived by Working Surface from the article; more in the Playbook. Source line: Right now code review is where those decisions get made, one PR at a time, by whoever is reviewing.

14 October 2026: as AI agents raise the volume of code changes, teams are moving routine checks into automated tests and installable rules, and keeping people for intent and contested decisions.