How Pinterest Improved Analytics Agent Compliance with Loop Engineering
Skyler Zheng, Chris Moradi, Lu Liu, Aman Gairola and Michael Chau · 12 November 2026
Read the originalPinterest engineers stopped hand-editing their data agent's instructions and built an automated loop that keeps only changes passing paired tests drawn from real chats.
Five Pinterest engineers, data scientists and managers described how they made the Analytics Agent, an AI agent that answers staff questions about company data, follow its required steps. The agent often skipped steps such as reading a table's notes before writing a query, and hand-written fixes to its instructions had unpredictable side effects. The team built tests from real chats and an automated loop that proposes small changes and keeps only those that pass.
- Even after manual rewrites, the agent read its data-discovery guide in only 64% of a broad set of chats. One sensible scope rule made it refuse a question it could answer.
- For each rule, the team built one test set of chats where the rule applies and one where it does not, so a fix cannot add unnecessary actions.
- Each change must fix more cases than it breaks, address a general pattern rather than one test case, and leave a larger regression suite intact. Changes are capped at a few files and lines.
- In identical repeat runs, 10.3% of cases changed result, so the team tracks which cases change rather than trusting small score gains.
The authors say people should set the goals, tests and limits while the loop proposes changes. A team tuning an AI agent could likewise turn real failures into tests and require each change to leave unrelated cases unchanged; that step is an inference.
When an engineering team changes an AI agent's instructions, test each rule against two sets of real chats, one where the rule should apply and one where it should not, and keep the change only if both pass.
After each change to an AI agent's instructions, rerun its tests and list every case that flipped between pass and fail, instead of trusting the total score.
Derived by Working Surface from the article; more in the Playbook. Source line: People define the goals, evaluation criteria, and safety boundaries.
13 November 2026: AI agents earn more freedom when a team limits what they can reach, from data to design components, and adds a check that does not depend on a glance.