Plays for engineering teams working with AI coding agents
What engineers and their managers can try, from reviewing AI-written code to the instruction files agents read, taken from what the teams in these articles did. Jev, an AI model from TypeSafe, checks that each one makes sense without its article. How the check works
Human approval
A person other than the one who built a piece of AI-assisted work checks it and approves it before it reaches customers, often called keeping a human in the loop.
Write down which actions an AI agent may take on its own and which wait for a person's approval.
From 7 articles · Amazon Web Services, PropCode, Salesforce and 4 moreShow all 7Hide
When a pull request was approved without every part being read, say in its description which parts nobody read, so that later reviewers and incident reviews know what was checked.
Before an AI agent's code change goes live, have an engineer check the agent's plan for watching it in use, and agree in advance what happens if a problem appears.
Assign the review of each agent-built code change to the tech lead or engineering manager of the team that owns that code, not to whoever is free.
Before reading an agent-written change line by line, rank its files by risk and spend close review on the riskiest ones.
Keep a fixed list of changes that always need a person's approval, including changes to the approval rules, and check it with a script the AI agent cannot edit.
Have a coding agent attach a note to each change listing what was asked, what changed, what has no test and what was left out, for the reviewer to check.
Log the median time to approve and the approval rate for each step where a person approves an AI agent's action. Redesign any step with a median under two seconds or approval above 95 percent.
Send every change that an AI agent makes through the same validation and saving code that people's changes use, so that the product's rules are kept in one place.
Report AI agent spending by team per piece of finished, reviewed work, so the cost of reviewing agent output counts, not only the AI model's bill.
Let the person who asked an AI agent for a code change review that change, and track how often people have to correct or redirect the agent.
Give every change an AI agent makes its own live preview link that people outside engineering can open.
When an AI agent drafts text for users, let the AI model write the wording, and let ordinary code route requests, copy record numbers and check the output format.
Review an AI agent's code by risk: a person for the risky changes, a lighter check for routine ones.
From 3 articles · OpenAI, GitHub and 1 moreShow all 3Hide
Agent context files
The files an AI agent reads before it works, such as AGENTS.md, CLAUDE.md, design-system rules and other written instructions, and the person or team who keeps them current.
Write down how senior engineers investigate a problem, including their checks, trusted signals and order of steps, and turn it into instructions an AI agent follows.
Put the rules an AI agent keeps getting wrong into the instruction file it reads every time.
From 4 articles · Linear, Assetnote, Salesforce and 1 moreShow all 4Hide
Test AI agents on the same tasks with and without their added instruction files, because extra context can confuse an agent and make its results worse.
Store the instructions every AI agent task reads, the files for one task and incoming streams such as email in three separate folders.
Keep every rule about who may see what in one central permissions table, so that adding a type of user or granting a permission is a one-line change.
Split agent work into steps that each map to one pull request, and run the coordination between steps in ordinary code rather than in an AI agent.
After each AI agent run, save what the agent had to learn, the decisions it made alone and the engineers' review comments, and update the reusable prompts from them.
Settle the design system before AI agents build screens, and point every agent at that one source.
From 4 articles · Aha!, SpaceXAI, Southleft and 1 moreShow all 4Hide
When AI agents write much of the code, invest in shared structure for the parts that change most, such as a design system or generated database queries.
Changing roles
How the jobs of product managers, designers, engineers and their managers change when AI agents do more of the building.
Report how long pull requests wait for review and how many are reworked afterwards, next to the amount of code produced.
Require the author of an AI-assisted pull request to explain the change, run the relevant tests and check security concerns before requesting a review.
State the team's high-level technical values to the coding agent explicitly before it starts work.
Set stricter automated checks for test coverage and code complexity on AI-written code than on hand-written code, and run them on every change before merging.
Read the code an AI model writes before it ships.
Cap how many topics each engineer and each squad may have open at once, and hold the cap when stakeholders or engineers push to start more AI-assisted work.
Compare every proposed ticket, design document or delegated task against the cost of making the change directly.
Agent ownership
A person or team answers for what an AI agent does, and decides its instructions, its default settings and when it must hand a decision to a person.
Count an AI agent's incident task as finished only when a separate check shows the problem has gone, such as the alarm clearing, not when the agent's action reports success.
Start using AI agents in one low-risk process that the rest of development does not depend on, and name one person in that team to set the agents up.
When AI agents repair their own failing work, cap the repair attempts and the spending, and hand the problem to an engineer once either limit is reached.
Give each AI coding agent one whole feature, from interface to storage, instead of one layer such as the frontend, so that agents seldom need to brief each other.
Have AI agents that check other agents' work answer pass, warn or fail in a fixed format, and compare several checkers' answers instead of trusting a single one.
When an AI agent takes over one low-risk development step, record the review and testing time of the steps before and after it. Compare that with the time saved on the step itself.
Evals before the build
Before an AI feature is built, the team writes down what a good output looks like, so that the result can be tested against it.
Have the AI agent ask its questions or show its plan before it builds anything.
From 4 articles · WitnessAI, Intent, QuantumBlack and 1 moreShow all 4Hide
Before trusting one AI model to grade another's output, label a set of examples by hand and measure how often the grader agrees.
Do not count tests that an AI model wrote for existing code as verification, because they record what the code already does, errors included.
Before an AI agent starts building, agree with the people who approve its changes which automatic tests every change must pass.
From 3 articles · Anthropic, GitHub Next, PikdShow all 3Hide
Training juniors
How people new to a craft build judgement when AI tools do the work that used to teach it.
Never assign a junior engineer a project alone, and have a senior engineer review the junior's pull requests, including code written with AI.
Run incident reviews that ask what the team believed that turned out to be untrue, so that junior engineers practise judging their own assumptions.
Teach junior engineers to review what an AI tool produces and to explain why some of its choices may cause problems.
Write critical thinking into career frameworks as specific behaviours, such as validating assumptions before implementation, so that managers can see a junior engineer's growth that shipped tickets no longer show.
Understanding the work
A team checks on purpose that its people can still explain work that AI agents produced.
Once AI agents write most of the code, have engineers present their finished work to the team, so that each keeps pride in the work and knows what colleagues build.
Ask the team that owns a critical service to explain its recent consequential changes without its go-to engineer or its AI tool before approving the next release.
Give the team that owns a critical service the authority to delay a release when it cannot explain a change, before any reliability target is breached.
Work cut to size
Before an AI agent starts, the work is split into pieces small enough for a person to review one at a time.
Run automated checks, such as behaviour tests and independent code reviews, before a person approves agent-built work, so that each piece reaches the person already checked.
Cut each agent-built feature into thin slices that each work end to end, and review the agent's implementation plan as a group before any code is written.
After the prototype
AI makes a working prototype quick, so the effort moves to choosing a direction and getting from prototype to production with a reliable product.
Other plays
Plays from links that are not filed under a topic.
When AI agents build the same feature separately for iOS and Android, make both versions pass one shared test suite for the business logic before either ships.
Require a test coverage check to pass before any pull request merges, because an AI coding agent may ignore a testing rule written in its instruction file.
Offer each new AI model to staff on release day as an experiment with a capped budget, and keep it only if benchmarks, user reports and cost data support it.
Compare a new AI model's cost per session on the same group of early users before and after, because early users are heavier users of AI than most staff.
Have the same developer build each mobile feature on both iOS and Android with AI coding agents, instead of keeping separate iOS and Android teams.
Sort pull requests by type each month and track features, automated checks and same-day emergency fixes side by side.
Before an AI model grades an AI agent's output, run a free check in ordinary code first, such as a rule that flags a list with too many items.