Working Surfacelink · 15 Nov 2026
03
A stonemason's curved wooden template hangs on a workshop wall while each newly carved stone block is held up against it, and out in the yard a figure seen from behind measures a weathered arch and carries a fresh template back toward the wall, the whole scene drawn with wide empty margins on every side.
Amazon Web Services

From prototype to production: Engineering agents with Agent DLC

Yuan Wei, Tianqin Cai and Jiade Wu · 12 November 2026

How-to · agents · decisions · Evals before the build

Read the original
Takeaway

Three Amazon Web Services authors propose that AI agents be released only against written criteria a business owner approves, and show one prompt change improving one answer while worsening another.

Summary

Yuan Wei and Tianqin Cai, applied scientists at Amazon Web Services (AWS), and Jiade Wu, an AWS architecture manager, ask when an AI agent is ready for production. They argue that unit tests are not enough, because an agent's answers vary between runs and shift when its prompts or outside services change. Their method, Agent DLC, judges every change against one set of criteria, thresholds and approved example answers that business owners sign off.

Key points
  • The authors built an assistant that answers staff questions about human resources policy. A score that judged retrieved passages whole rated every answer perfect, yet for a sick-leave question about nine in ten retrieved facts were irrelevant.
  • Changing only the system prompt improved one answer and made the sick-leave answer worse. The documents held no sick-leave policy, so no prompt change could supply the answer.
  • Each criterion sits in one of three tiers. A hard constraint blocks release on any failure, a release gate needs a pass rate over repeated runs, and other measures are only monitored.
  • Work on the agent happens against fixed criteria. The criteria themselves change only when production use reveals a case none of them covered, so a missing requirement is not patched with a prompt change.
Implication

A team building an AI agent can have its business owner approve 20 to 30 example requests and their expected answers before the first evaluation, as the authors advise. The post is a vendor's account: AWS sells the Amazon Bedrock AgentCore services it recommends, and the example is the authors' own test system, not a customer result.

Plays
Engineering

Have the business owner approve 20 to 30 example requests and their expected answers before measuring an AI agent's first baseline.

Engineering

When an AI agent fails on a case that no criterion covers, add the criterion and its examples first, then change the agent to meet them.

Derived by Working Surface from the article; more in the Playbook. Source line: Confusing the two can lead teams to try to fix a missing criterion with a prompt change.

Source issue

15 November 2026: A team can rely on an AI agent's work only when the check sits outside the agent and someone other than the builder decides what correct means.