From prototype to production: Engineering agents with Agent DLC
Yuan Wei, Tianqin Cai and Jiade Wu · 12 November 2026
Read the originalThree Amazon Web Services authors propose that AI agents be released only against written criteria a business owner approves, and show one prompt change improving one answer while worsening another.
Yuan Wei and Tianqin Cai, applied scientists at Amazon Web Services (AWS), and Jiade Wu, an AWS architecture manager, ask when an AI agent is ready for production. They argue that unit tests are not enough, because an agent's answers vary between runs and shift when its prompts or outside services change. Their method, Agent DLC, judges every change against one set of criteria, thresholds and approved example answers that business owners sign off.
- The authors built an assistant that answers staff questions about human resources policy. A score that judged retrieved passages whole rated every answer perfect, yet for a sick-leave question about nine in ten retrieved facts were irrelevant.
- Changing only the system prompt improved one answer and made the sick-leave answer worse. The documents held no sick-leave policy, so no prompt change could supply the answer.
- Each criterion sits in one of three tiers. A hard constraint blocks release on any failure, a release gate needs a pass rate over repeated runs, and other measures are only monitored.
- Work on the agent happens against fixed criteria. The criteria themselves change only when production use reveals a case none of them covered, so a missing requirement is not patched with a prompt change.
A team building an AI agent can have its business owner approve 20 to 30 example requests and their expected answers before the first evaluation, as the authors advise. The post is a vendor's account: AWS sells the Amazon Bedrock AgentCore services it recommends, and the example is the authors' own test system, not a customer result.
Have the business owner approve 20 to 30 example requests and their expected answers before measuring an AI agent's first baseline.
When an AI agent fails on a case that no criterion covers, add the criterion and its examples first, then change the agent to meet them.
Derived by Working Surface from the article; more in the Playbook. Source line: Confusing the two can lead teams to try to fix a missing criterion with a prompt change.
15 November 2026: A team can rely on an AI agent's work only when the check sits outside the agent and someone other than the builder decides what correct means.