Working Surfacelink · 14 Nov 2026
03
A tall wooden card cabinet with every drawer kept full stands over a wide sieve tray, through which only a few cards weighted with heavy lead tags drop into a basket that a figure seen from behind carries away, the whole scene drawn with wide empty margins on every side.
PIVOT Tech Blog

A record of choosing which ever-growing AI-written test cases to run, using risk-based testing

Keisuke · 12 November 2026

How-to · critique & review · decisions · Decision records

Read the original
Takeaway

A quality assurance engineer at PIVOT halved the test cases run before each release by ranking them by harm, after AI-written cases outgrew manual release testing.

Summary

Keisuke, a quality assurance engineer at PIVOT, a Japanese business video company, published an account in Japanese of how the team chooses which test cases to run before release. PIVOT writes test cases with AI from product documents, Figma designs and code changes, and the cases double as its specification. They grew faster than people could run them in acceptance testing, once reaching 277 for one release.

Key points
  • Each acceptance criterion gets one of four harm levels, from a viewer being unable to watch a video to an add-on feature failing. It also gets a likelihood based on whether the release changes it.
  • A table, not a numeric score, sets how many cases run for each combination, so the team can explain the remaining risk to the product manager.
  • A Claude Code skill gathers the release's pull requests, confirms each likelihood with the QA engineer, selects cases and writes the run record in Notion. A written rule document overrides the skill when they disagree.
  • For a large restructuring release, the cases to run fell from 1,339 to 628, and the time saved went to exploratory testing of high-harm features.
Implication

The author advises teams to rate harm per behaviour rather than per feature, and to try any new selection rule on past releases and their bugs before using it.

Plays
Engineering

Record every test case left out of a release with the reason it was left out, so the choice can be reviewed after a bug.

Engineering

Before a release, choose test cases with a written table that rates each acceptance criterion by how badly its failure would harm users and whether the release changes it.

Source issue

14 November 2026: as AI agents produce more code and tests than people can inspect, teams are writing down which work gets checked closely, by whom or by what, at what cost, and how they will know it worked.