Working Surfacelink · 7 Nov 2026
02
A clerk seen from behind ignores a small brass bell rung on a hotel front desk and instead pulls open a long drawer of index cards, holding each card against a master card to see whether they match, the whole scene drawn with wide empty margins on every side.
Microsoft Command Line

The check that catches agent failures can also train them

Tuhin Kundu and Mirco Milletari · 6 November 2026

How-to · critique & review · agents · Ground truth for review

Read the original
Takeaway

Microsoft scientists found that most failed AI agent runs looked complete, so they grade AI agents on the records left behind and count tasks passed every time.

Summary

Tuhin Kundu and Mirco Milletari, applied scientists at Microsoft, built ThinkingBox, an open test framework for AI agents that change business records. Their second post on it reports a test of 18 AI models on 507 synthetic tasks, each run 20 times. They also used the same check to train an open-weight AI model.

Key points
  • About two in three failed runs ended with the AI agent signalling completion and no visible error, yet the records did not match what was expected.
  • Counting tasks passed on all 20 attempts ranked AI models differently from the average pass rate, and changed which AI model was cheapest per dependable task.
  • Training the open-weight Qwen3.8-27B with the same check raised its tasks passed every time from 38 to 107, and cut its cost per dependable task from $24.36 to $8.00.
  • The authors note limits: the tasks are synthetic reconstructions of business work, and a check rewards only what its specification requires.
Implication

An engineering team that lets an AI agent change production records can grade it on the resulting records and on repeated runs, as the authors advise. The authors are Microsoft scientists reporting their own framework and test results.

Plays
Engineering

Grade an AI agent by checking the records it leaves behind, not by reading its own report of what it did.

Engineering

When testing an AI agent, run each test task many times and report how many tasks it passed on every attempt beside its average pass rate.

Derived by Working Surface from the article; more in the Playbook. Source line: By those signals, which a transcript or tool-call log can show, about two in three failed runs looked complete.

Source issue

7 November 2026: teams using AI agents are judging work by evidence they can check, and are naming the people who decide what counts as good.