Making Failure Diagnosis Reliable with Pensive AI

Netflix engineers added an AI model to their failure-diagnosis system, using old hand-written rules as reference answers and evaluation scores to gate every merge.

Netflix, the streaming service, runs millions of data and machine-learning jobs a day, and roughly 100,000 fail on peak days. Its rule-based diagnosis system, Pensive, could not classify about half of the failures it processed. Praneeth Yenugutala and five colleagues on Netflix's data platform describe Pensive AI, which adds an AI model while keeping the rules that already worked.

• The AI model returns only the first and last line numbers of an error, so the text shown comes from the original log and costs less to produce.
• Old hand-written rules provide reference answers, and two AI judges score whether known errors still appear and whether each extracted error is useful.
• Each pull request runs the evaluations in a preview environment and must pass precision and inclusion thresholds to merge; a failing promoted build alerts the on-call engineer.
• Every thumbs-down on a diagnosis sends a Slack alert during business hours, linked to the full record of how the AI reached its answer.

An engineering team adding AI to an existing rule-based system can keep the rules as a floor that new versions must not fall below, and gate merges on evaluation scores. Netflix calls its main usage figure observational, not causal.

Derived by Working Surface from the article; more in the Playbook. Source line: Precision and inclusion thresholds gate the merge.

27 October 2026: AI produces reliable work when a team writes down how its own system works and checks the output against something the AI did not write, from frozen rules to audits.