Beyond the model: Engineering AI infra with scientific judgement

Airbnb built Insight Miner, a tool that writes its analysis method into software around an AI agent, so that findings from customer-support data can be reproduced and checked.

Before launching an AI customer service assistant, Airbnb needed to know what situations it would face, including rare risky ones, from past support conversations. Each such investigation took months of hand work, and the team soon needed one almost every week. Wren Dougherty, writing on Airbnb's tech blog, describes Insight Miner, the tool Airbnb built so that an AI agent carries out the analysis with a shared method.

• Dougherty warns that an AI agent asked to analyse 100,000 support conversations returns a polished answer but hides how it reached it.
• Insight Miner runs the mechanical steps: finding the data, labelling it and grouping similar conversations. People spend their time on the most ambiguous cases and on testing hypotheses.
• Investigations that took months now take days. A year in, dozens of teams use it, and most users work outside technical roles, mainly in operations and product research.
• Separate AI agents maintain the tool, updating its instructions for new models and proposing fixes for bugs.

Writing the method of an analysis into a shared tool lets others reproduce and challenge the results. Airbnb makes this claim for data science, and its application to other agent-built work is an inference.

Derived by Working Surface from the article. Source line: results can be reproduced, audited, and challenged.

18 September 2026: a review of an agent's work is only as good as the reference behind it.