Plays for product managers working with AI agents
What product managers and product owners can try, taken from what the teams in these articles did once AI agents took on more of the building. Jev, an AI model from TypeSafe, checks that each one makes sense without its article. How the check works
Decision records
A written record of what was decided and why, kept with the work so that someone who did not build it can review it.
Keep a visible list on the team's planning board of each customer problem the team chose not to solve and each solution it rejected.
Replace a feature's requirements document and design brief with one table that has a row for each thing the product lets someone do. Link a click-through prototype under each row, and have a designer record a video walkthrough at the top.
Put a short human-written brief at the top of every ticket or specification handed to an agent, and label each section as written by a person or generated by the agent.
List the tasks AI agents now do, have their former owners test whether they can still do them unaided, and schedule practice without AI for skills that are slipping.
Link every synthesised finding and ticket to the transcript or feedback it came from, and stop work on any item that cannot be traced to its source.
Give a named person time to keep the documents that explain the product current, and to mark the ones that are out of date.
Agent context files
The files an AI agent reads before it works, such as AGENTS.md, CLAUDE.md, design-system rules and other written instructions, and the person or team who keeps them current.
Make one person responsible for the design system, separate from whoever builds screens, and have that person review every gap or exception the builders record.
Give an AI agent one new kind of task at a time, such as testing its own work before merging, and add the next only once it handles that reliably.
On a schedule, read in full the instruction file that an AI agent loads for every task, and move task-specific instructions into separate files.
Check which shared files and systems a colleague's AI assistant can reach before concluding that the colleague resists using it.
Settle the design system before AI agents build screens, and point every agent at that one source.
From 4 articles · Aha!, SpaceXAI, Southleft and 1 moreShow all 4Hide
After the prototype
AI makes a working prototype quick, so the effort moves to choosing a direction and getting from prototype to production with a reliable product.
Release a new feature to half of new users and keep the other half unchanged as a comparison group.
Show the working prototype to stakeholders before the production build, so that engineering and marketing partners can commit early.
Write down the business rules a piece of software must follow before prototyping it with AI, and give developers those rules along with the prototype.
Revise the written specification and the working prototype against each other, so that a change in one is checked in the other.
Before a test starts, write down how many users it needs and the date the result will be read.
Human approval
A person other than the one who built a piece of AI-assisted work checks it and approves it before it reaches customers, often called keeping a human in the loop.
Write down which actions an AI agent may take on its own and which wait for a person's approval.
From 7 articles · Amazon Web Services, PropCode, Salesforce and 4 moreShow all 7Hide
Ask whoever hands over work that AI helped produce to confirm they have checked all of it, and treat any error in it as their error.
Publish the list of checks an agent must pass before release at the start of a project, so that builders plan for them from the first day.
Before a person reviews an AI agent's answers, give that person a verified reference to check them against, and re-check any corrections the reviewer makes.
Evals before the build
Before an AI feature is built, the team writes down what a good output looks like, so that the result can be tested against it.
Have the AI agent ask its questions or show its plan before it builds anything.
From 4 articles · WitnessAI, Intent, QuantumBlack and 1 moreShow all 4Hide
Have a person read 10 real user sessions with an AI feature and note what went wrong, then 100, before anyone writes a measure of its quality.
Before an AI agent starts building, agree with the people who approve its changes which automatic tests every change must pass.
From 3 articles · Anthropic, GitHub Next, PikdShow all 3Hide
Changing roles
How the jobs of product managers, designers, engineers and their managers change when AI agents do more of the building.
Before pausing work done with AI agents, share its files, write down the next step and list open questions, so a colleague can continue it without the person who started.
Have product leaders record, for each product, whether its product managers and designers may deploy their own changes to production, rather than one rule for all teams.
Agent ownership
A person or team answers for what an AI agent does, and decides its instructions, its default settings and when it must hand a decision to a person.
Make one person responsible for how an AI-built product fits together as a whole, and have that person check that each new change fits.
Give every instruction file that an AI agent works from a named owner who keeps it current.
From 3 articles · WitnessAI and 2 moreShow all 3Hide
Ground truth for review
A set of correct answers or expected results, prepared by someone other than the builder, that a reviewer uses to check work made with AI.
Write down the problem a feature solves before building it, and judge the result against that statement.
From 2 articlesShow all 2Hide
When an AI agent analyses data, save the steps it followed in a tool the whole team can use, so that anyone can repeat the analysis and check the answer.
Understanding the work
A team checks on purpose that its people can still explain work that AI agents produced.
Interfaces for agents
Product pages and data designed so that an AI agent, not only a person, can use them.
Work cut to size
Before an AI agent starts, the work is split into pieces small enough for a person to review one at a time.
Other plays
Plays from links that are not filed under a topic.
Ask users what they expect before they first use an AI feature, then compare that with what they report afterwards.
Add the question of whether users understand and trust a feature to every product review, beside the question of whether it works.
Design an AI feature to hand users the finished result first, such as a personalised report, and offer the underlying tool and controls as a second step.