The more elaborate the code review skill, the less often its comments were fixed: results of six months of checking
Anago · 15 November 2026
Read the originalA developer recounted six months of AI pull-request reviews and found that the most elaborate review skill had the smallest share of comments acted on, so kept only the simplest.
Anago, a developer writing on the Japanese site Zenn, published an account in Japanese of six months of AI code review on a mobile app and its backend. Anago had written three skills, sets of instructions that tell an AI agent how to review a pull request. Anago then recounted all their comments under one fixed definition of a fix.
- Anago counted 1,306 comments from April to 13 November 2026, on merged pull requests whose authors had replied, and treated a comment as fixed only if the code then changed.
- The basic line-by-line skill had 68.9% of its comments fixed, at 4.4 comments per pull request. The elaborate skill, which ran 11 to 14 viewpoints in parallel, had 59.2% fixed, at 7.8 per pull request.
- All three skills were wrong on about one comment in ten, so the error rate alone could not separate them. The elaborate skill raised more points that were correct but low priority or outside the pull request's scope.
- Anago now runs only the basic skill on a fixed schedule, stopped a third skill whose comments were only cosmetic, and recounts each month with the same script.
A team comparing AI review tools can read two numbers together: the share of comments that led to a code change, and the number of comments per pull request. This is an inference from one developer's own analysis. In that analysis, 36% of comments could not be traced to a skill, and a spot check found 2 of 8 "fixed" judgements were unrelated changes.
Define once what counts as an AI review comment being fixed, such as the code changing afterwards, and recount every review tool against that definition each month.
When comparing AI code review tools, report each tool's share of comments that led to a code change next to its number of comments per pull request.
16 November 2026: A team learns what AI is worth by counting what its people kept from its work, by having experts correct what agents read, and by recording where work waits.