Experimentation, AI and the Product Model

Marty Cagan of Silicon Valley Product Group argues that cheap AI building makes choosing what to build the hard part, so product teams must own experiments with expert coaching.

Marty Cagan, founder of the product management consultancy Silicon Valley Product Group, wrote this essay from a keynote at an experimentation conference in Austin. He argues that AI has made building faster and cheaper, yet many AI launches have frustrated customers and few have improved results. He sets out how product organisations should run experiments now that choosing what to build is the hard part.

• Cagan separates three kinds of product work: new products, new value in existing products, and optimisation of existing value, and each needs different experiments.
• Most companies limit experiments to optimisation and split them across separate research, data science and conversion teams.
• Cagan recommends a hybrid model in which product teams own their experiments and experimentation specialists coach them, because teams take findings more seriously when they run the tests.
• For AI products, Cagan says teams should judge quality with evals early and keep A/B tests for establishing value, since those tests are slower and their results last less long.

A product organisation can move its research and data specialists into a coaching role inside product teams instead of a queue of requests. Cagan sells product coaching, and the essay is his argument, not a measured study.

Derived by Working Surface from the article; more in the Playbook. Source line: If you want to innovate, you simply must experiment.

22 October 2026: as AI agents make building cheap, teams are moving effort to deciding what to build and writing down the reasons where people and AI agents can reuse them.