A record of choosing which ever-growing AI-written test cases to run, using risk-based testing
Keisuke · 12 November 2026
Read the originalA quality assurance engineer at PIVOT halved the test cases run before each release by ranking them by harm, after AI-written cases outgrew manual release testing.
Keisuke, a quality assurance engineer at PIVOT, a Japanese business video company, published an account in Japanese of how the team chooses which test cases to run before release. PIVOT writes test cases with AI from product documents, Figma designs and code changes, and the cases double as its specification. They grew faster than people could run them in acceptance testing, once reaching 277 for one release.
- Each acceptance criterion gets one of four harm levels, from a viewer being unable to watch a video to an add-on feature failing. It also gets a likelihood based on whether the release changes it.
- A table, not a numeric score, sets how many cases run for each combination, so the team can explain the remaining risk to the product manager.
- A Claude Code skill gathers the release's pull requests, confirms each likelihood with the QA engineer, selects cases and writes the run record in Notion. A written rule document overrides the skill when they disagree.
- For a large restructuring release, the cases to run fell from 1,339 to 628, and the time saved went to exploratory testing of high-harm features.
The author advises teams to rate harm per behaviour rather than per feature, and to try any new selection rule on past releases and their bugs before using it.
Record every test case left out of a release with the reason it was left out, so the choice can be reviewed after a bug.
Before a release, choose test cases with a written table that rates each acceptance criterion by how badly its failure would harm users and whether the release changes it.
14 November 2026: as AI agents produce more code and tests than people can inspect, teams are writing down which work gets checked closely, by whom or by what, at what cost, and how they will know it worked.