Working Surfaceplaybook

Plays for engineering teams working with AI coding agents

What engineers and their managers can try, from reviewing AI-written code to the instruction files agents read, taken from what the teams in these articles did. Jev, an AI model from TypeSafe, checks that each one makes sense without its article. How the check works

295 plays from 194 links

Human approval

A person other than the one who built a piece of AI-assisted work checks it and approves it before it reaches customers, often called keeping a human in the loop.

68 plays · Topic pageClearest first, as Jev scored them
CHECKED BY JEV · WORKING SURFACE ·22 NOV2026
Engineering

When prompting an AI coding agent for a change, tell it which requirements the change can ignore, for example that cached data may be briefly out of date, so it does not add unneeded locks and checks.

seangoedecke.com · What requirements don't you have? · 20 Oct 2026
Engineering

Assign the review of each agent-built code change to the tech lead or engineering manager of the team that owns that code, not to whoever is free.

Design Outcomes · The Bottleneck Moved to Review · 21 Sep 2026
Engineering

Require screenshots or short recordings of the state before and after a change in every pull request an AI agent opens.

Warp · Addressing the code review bottleneck · 8 Nov 2026
Engineering

When an AI assistant maps or writes code for a system with no documentation, have an engineer review everything it produces, and keep telling it what the task is and why it matters.

LeadDev · The reality of being a forward deployed engineer (FDE) · 17 Oct 2026
Engineering

Write a specification of the problem and the required behaviour before asking a coding agent to build an internal tool.

Salesforce Engineering · How AI Slashed 600-Test-Failure Triage From Days to 90 Minutes · 18 Nov 2026
Engineering

Sort AI-written pull requests by risk: let test and documentation changes merge once automated checks pass, and send authentication, billing and public API changes to an engineer qualified to review them.

Dev Interrupted · All the code review is a stage · 14 Nov 2026
Engineering

Let AI agents build and deploy changes to test environments, but require a person's approval before any change merges to the main branch or reaches production.

Salesforce Engineering · Scaling AI Engineering Automation Without Sacrificing Reliability · 10 Nov 2026
Engineering

Keep a list of every AI agent in use, with who uses it and what it may change, and give each agent only the permissions it needs.

Forbes Technology Council · Agentic AI Is Shipping Code. Who Owns The Risk? · 10 Nov 2026
Engineering

When a pull request was approved without every part being read, say in its description which parts nobody read, so that later reviewers and incident reviews know what was checked.

Elevate · The Code Nobody Reads · 28 Sep 2026
Engineering

Tag every pull request by type, such as feature, bugfix or maintenance, and track the mix instead of the total count.

The Pragmatic Engineer · Uber's AI infra, from the inside · 16 Oct 2026
Engineering

Let an AI agent investigate production problems with read-only access, then require approval of its full plan before it gets write access.

Vercel · How Vercel Agent accesses production safely · 24 Oct 2026
Engineering

Require an approver other than the engineer who asked for it on every pull request an AI agent opens, and enforce the rule in branch protection.

A.Team · Agentic engineering for teams: What changes when every engineer runs agents · 3 Nov 2026
Engineering

Limit the coding agent to safe, well-scoped customer changes and have it open a pull request with a short spec and screenshots for an engineer to review.

Vercel · How Rillet ships 3× faster with AI agents on Vercel · 11 Oct 2026
Engineering

Before reading an agent-written change line by line, rank its files by risk and spend close review on the riskiest ones.

JetBrains Research · Our Framework for Reviewing AI-Generated Code · 7 Oct 2026
Engineering

Keep changes to migrations, sign-in, permissions, billing and deployment settings under human review whatever score an AI reviewer gives them.

Rootly · When should AI be allowed to approve a pull request? · 19 Nov 2026
Engineering

Treat the review of every new dependency as part of code review, including for pull requests written by AI.

LeadDev · Engineers lose 17+ days a year to AI rework · 21 Oct 2026
Engineering

After a production incident, stop pull requests of the same kind from merging on automated checks alone until the team fixes the check that missed the problem.

Dev Interrupted · All the code review is a stage · 14 Nov 2026
Engineering

Write down who may open a pull request, who reviews it and what a change from outside engineering must prove, before adding more AI tools.

AutonomyAI · The bottleneck moved: what 131 product and engineering teams told us · 20 Nov 2026
Engineering

Turn each bug that caused an incident into an automated check, and require an escalation with a written reason to bypass any check.

InfoQ · Move Fast, Don’t Break Trust: Shipping Constantly with Humans and Beyond · 25 Oct 2026
Engineering

Before an AI agent's code change goes live, have an engineer check the agent's plan for watching it in use, and agree in advance what happens if a problem appears.

Cursor · Bots for the last mile: Rollouts, Security Review · 24 Sep 2026
Engineering

Have a person approve the AI agent's implementation plan before any code is written, and post the approved plan on the issue for the next person or agent.

Thoughtworks · How Sensible Harness can help teams create an AI-first delivery environment · 23 Oct 2026
Engineering

Test changes to an AI agent's investigation instructions by replaying past production incidents, with the answer hidden from the agent.

OpenAI · How OpenAI’s Performance Factory keeps ChatGPT fast · 21 Nov 2026
Engineering

After three rounds of AI review and fixes on one pull request, have the AI reviewer stop and ask the engineer who wrote it to rethink the approach.

claude.dev Blog · How the Claude Code team does code review · 18 Oct 2026
Engineering

When reviews are slower than usual, have the AI agent request reviews for the engineer's waiting pull requests, with the engineer's approval, before starting new work.

Dev Interrupted · Your agents should ask the factory first · 17 Oct 2026
Engineering

Send every change that an AI agent makes through the same validation and saving code that people's changes use, so that the product's rules are kept in one place.

Dropbox.Tech · Evolving our calendar assistant Reclaim to be AI-native without starting over · 30 Sep 2026
Engineering

Label every code change an AI agent writes, then compare how often those changes contain security flaws, and how often reviewers catch problems before merge, with changes people wrote.

Forbes Technology Council · Agentic AI Is Shipping Code. Who Owns The Risk? · 10 Nov 2026
Engineering

Mark which parts of the codebase AI agents may change with light review and which need a named person to read every change, and make the repository require that person's approval.

A.Team · Agentic engineering for teams: What changes when every engineer runs agents · 3 Nov 2026
Engineering

Tag every ticket that fixes AI-assisted code with an AI rework label in the issue tracker, so the hours spent on that rework show in sprint reports.

LeadDev · Engineers lose 17+ days a year to AI rework · 21 Oct 2026
Engineering

Review a release that bundles several pull requests, together with its plan, instead of reading each pull request alone.

The Pragmatic Engineer · Software factories and code reviews with Harjot Gill · 19 Nov 2026
Engineering

Post a cost report to the ticket after an AI agent finishes each task, so the team can see where the spending goes.

Thoughtworks · How Sensible Harness can help teams create an AI-first delivery environment · 23 Oct 2026
Engineering

Let named engineers release code to production without review, and put their releases back under review after a set number of outages traced to their changes.

The Pragmatic Engineer · Software factories and code reviews with Harjot Gill · 19 Nov 2026
Engineering

Have the AI agent score each pull request for how well it was tested, what it touches and how hard it is to undo, so the reviewer knows where to look first.

Warp · Addressing the code review bottleneck · 8 Nov 2026
Engineering

Before an AI agent starts a new task, compare how long the team's pull requests now wait for a first review with the usual wait.

Dev Interrupted · Your agents should ask the factory first · 17 Oct 2026
Engineering

When an AI agent writes a bug fix, have a different AI model review it, and have engineers check and sign off on those reviews across many incidents.

Salesforce Engineering · Scaling AI Engineering Automation Without Sacrificing Reliability · 10 Nov 2026
Engineering

Send engineers only the highest-ranked bug reports from AI-written tests, after AI models and fixed rules remove false alarms, and ask whether each change in behaviour was intended.

Engineering at Meta · Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale · 1 Nov 2026
Engineering

Sort each pull request from an AI agent by risk, have an engineer review the risky ones and give routine ones a lighter check.

From 3 articles · OpenAI, GitHub and 1 moreShow all 3Hide
OpenAI · in The Pragmatic Engineer · 17 Sep 2026Each agent-written change is sorted by risk, and a person reviews the high-risk ones.
GitHub · in GitHub Blog · 21 Sep 2026The reviewer of risky code keeps going until they can explain the change and take responsibility for it.
Elevate · 28 Sep 2026Agents review every pull request, careful human review is kept for core and sensitive code, and a person approves every merge.
Engineering

Assign routine clean-up tasks to an internal AI agent through a chat message and let it gather its own context from earlier team conversations.

Sentry Blog · We have a Slopware Factory · 13 Nov 2026
Engineering

Keep a fixed list of changes that always need a person's approval, including changes to the approval rules, and check it with a script the AI agent cannot edit.

Medium · The Approve Button Is Not a Control · 29 Sep 2026
Engineering

When a production failure exposes a rule the code does not show, write it in the instruction file that engineers and AI coding agents read before changing code.

SoluteLabs · The Agentic Harness: How We Rebuilt Our SDLC for AI-Augmented Product Engineering · 9 Oct 2026
Engineering

Require review by a named human code owner of the service on every production change, and let automated checks confirm the code is built correctly.

InfoQ · Move Fast, Don’t Break Trust: Shipping Constantly with Humans and Beyond · 25 Oct 2026
Engineering

Have a second AI agent session try to disprove an AI agent's diagnosis before any fix is written.

OpenAI · How OpenAI’s Performance Factory keeps ChatGPT fast · 21 Nov 2026
Engineering

After each bug in code written by AI agents, name the class of error, add an automated check that detects it, then change the code so that error cannot be written again.

Accidentally in Code · Escaping the No-Code Platform by Building a Software Factory · 28 Oct 2026
Engineering

Rank each part of the code base by the harm a mistake there would cause, and require a human merge for agent changes only in the riskiest parts.

Auth0 Blog · Agentic Development: Time to Shift the Focus · 20 Oct 2026
Engineering

Have a subject-matter expert check any table mapping that an AI assistant produces before it is used in a migration.

LeadDev · The reality of being a forward deployed engineer (FDE) · 17 Oct 2026
Engineering

When reviewing an AI-generated change, look first for added locks, extra checks and layers that no requirement calls for.

seangoedecke.com · What requirements don't you have? · 20 Oct 2026
Engineering

Write down approval tiers by risk, set from what a change touches rather than who wrote it, and require two people only for the riskiest paths.

claude.dev Blog · How the Claude Code team does code review · 18 Oct 2026
Engineering

Turn the code review rules that nobody on the team disputes into lint rules, tests and automated checks that run before a person reviews.

Accidentally in Code · Attempting to reverse engineer code review · 14 Oct 2026
Engineering

Have a separate AI review pass check each larger AI-written change before a human reviewer sees it.

SoluteLabs · The Agentic Harness: How We Rebuilt Our SDLC for AI-Augmented Product Engineering · 9 Oct 2026
Engineering

Have code, not the AI model, verify each required approval for a production change, and let the AI model only explain the result.

Salesforce Engineering · How an AI Agent Cut Production Change Reviews From 24 Hours to 12 Seconds · 22 Nov 2026
Engineering

Have a coding agent attach a note to each change listing what was asked, what changed, what has no test and what was left out, for the reviewer to check.

Medium · The Approve Button Is Not a Control · 29 Sep 2026
Engineering

Have an AI agent that reviews production code changes pass a review to a person whenever it cannot find a required approval in the approval records.

Salesforce Engineering · How an AI Agent Cut Production Change Reviews From 24 Hours to 12 Seconds · 22 Nov 2026
Engineering

Report AI agent spending by team per piece of finished, reviewed work, so the cost of reviewing agent output counts, not only the AI model's bill.

Agoda Engineering & Design · Key Takeaways from Agoda’s AI Developer Report 2026 · 29 Sep 2026
Engineering

Move simple checks such as linting onto the engineer's machine, so that AI agents run them before opening a pull request.

The Pragmatic Engineer · Uber's AI infra, from the inside · 16 Oct 2026
Engineering

Let an AI review tool approve a pull request without a human reviewer only when it is under 1,000 lines, has no serious findings and its author has ten merged changes in that service.

Irrational Exuberance · Speeding up code review with small-change · 18 Oct 2026
Engineering

Require a review on the default branch so that no unreviewed change, from a person or an AI agent, can be merged.

The GitHub Blog · Building Git infrastructure for agent-scale development · 8 Oct 2026
Engineering

Let engineers merge routine changes without a second reviewer when they accept responsibility for fixing them, and keep reviews for the core of the product.

InfoQ · Doing The Right Thing: Accountability In The Age Of AI Coding Summary · 15 Nov 2026
Engineering

When turning code review comments into automated checks, hold back any rule reviewers disagree on until the team agrees it and names who decides.

Accidentally in Code · Attempting to reverse engineer code review · 14 Oct 2026
Engineering

For each step where a person approves an AI agent's action, log the median time to approve and the approval rate. Redesign any step with a median under two seconds or an approval rate above 95 percent.

UX Tigers · User Vigilance Fails Twice in the AI Age: Watch Duty & Verdict Duty · 1 Oct 2026
Engineering

Each night, have AI models write tests for the previous day's riskiest code changes, chosen by a risk score, so that a test fails if a change introduced a bug.

Engineering at Meta · Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale · 1 Nov 2026
Engineering

Give every change an AI agent makes its own live preview link that people outside engineering can open.

Vercel · How Delphi ships 100 times a day with its Python backend on Vercel · 17 Sep 2026
Engineering

Let the person who asked an AI agent for a code change review that change, and track how often people have to correct or redirect the agent.

Lenny's Newsletter · How I AI: Meta's Muse review + How Warp ships 2,000 PRs a month with AI factories · 22 Sep 2026
Engineering

Give an AI agent short-lived credentials limited to the approved plan and to the approving person's own permissions.

Vercel · How Vercel Agent accesses production safely · 24 Oct 2026
Engineering

Make every new commit cancel earlier approvals, so each merged pull request carries an approval of its final version.

Rootly · When should AI be allowed to approve a pull request? · 19 Nov 2026
Engineering

Let ordinary code fetch, group and reassign test failures, use an AI model only for unfamiliar errors, and have an engineer approve each conclusion.

Salesforce Engineering · How AI Slashed 600-Test-Failure Triage From Days to 90 Minutes · 18 Nov 2026
Engineering

When an AI agent drafts text for users, let the AI model write the wording, and let ordinary code route requests, copy record numbers and check the output format.

Salesforce Engineering · How Deterministic Controls Turn AI Output Into Reliable Prompt Templates · 2 Oct 2026
Engineering

Stop AI agents from merging database migrations, so that a person takes responsibility for each one.

Accidentally in Code · Escaping the No-Code Platform by Building a Software Factory · 28 Oct 2026
Engineering

When a small pull request that an AI review tool approved alone causes an error, tighten the team's eligibility rules for such approvals instead of blaming its author.

Irrational Exuberance · Speeding up code review with small-change · 18 Oct 2026

Changing roles

How the jobs of product managers, designers, engineers and their managers change when AI agents do more of the building.

47 plays · Topic pageClearest first, as Jev scored them
WORKING SURFACECHECKEDBY JEV · 22 NOV 2026
Engineering

Pair each new engineer with an experienced engineer who coaches them on when to trust and when to question work from AI tools.

The Pragmatic Engineer · Software engineering after code becomes cheap with Satya Nadella · 17 Nov 2026
Engineering

Have AI tools write the team's status updates and progress reports, and spend the engineering manager's freed time on coaching and on aligning priorities with dependent teams.

LeadDev · Long live the engineering manager · 20 Oct 2026
Engineering

Engage a UX designer to produce the designs that will be built, even when the design document already holds AI-generated mock-ups.

LeadDev · AI turned tech leads into architects · 4 Nov 2026
Engineering

When an engineer uses an AI model to propose changes outside their own area, have the domain expert for that area review the design document before work starts.

LeadDev · AI turned tech leads into architects · 4 Nov 2026
Engineering

Have engineers build a separate system that checks an AI agent's work, including that no failing test was deleted, instead of accepting the agent's own report of success.

The Pragmatic Engineer · Software engineering after code becomes cheap with Satya Nadella · 17 Nov 2026
Engineering

When reviewing code an AI agent wrote, question each added safeguard, such as a new lock on shared data, and ask which real requirement it serves.

seangoedecke.com · Getting good at AI takes time and practice · 29 Oct 2026
Engineering

Share department-wide AI usage figures with every engineer, show names only to each engineer's own manager, and never rank engineers by usage.

LeadDev · Cheap AI tokens are running out · 30 Oct 2026
Engineering

Write the company's rules for installing Model Context Protocol servers, the connectors that give AI agents access to data and tools, so no engineer installs one straight from the internet.

LeadDev · Software developers are becoming agent builders · 10 Oct 2026
Engineering

State the team's high-level technical values to the coding agent explicitly before it starts work.

Sean Goedecke · Human-AI partnerships are for alignment, not capability · 3 Oct 2026
Engineering

Have the AI coding agent produce a short readable summary of each pull request it opens, for the engineer to read before merging.

The Pragmatic Engineer · How software engineering is changing: a tale of two eras · 4 Nov 2026
Engineering

Run a week-long trial in which AI agents write all the code, and judge it by what the team learns, not by how much code the agents produced.

martinfowler.com (Rachel's Ramblings) · Why are good developers so wary of agentic development? · 30 Oct 2026
Engineering

Run an AI agent on the same job twice, once with its stored memory and once without, and compare speed and quality to see whether the memory helps.

Microsoft Command Line · The new primitives of agentic software delivery · 31 Oct 2026
Engineering

Add task abandonment, conversion drops and support ticket spikes as signals in the monitoring of deployed software, so that a fall in user success triggers the same automated response as a spike in server errors.

Nielsen Norman Group · The Role of UX in Agentic Software Factories · 31 Oct 2026
Engineering

Require the author of an AI-assisted pull request to explain the change, run the relevant tests and check security concerns before requesting a review.

LeadDev · Faster code isn’t faster delivery · 7 Oct 2026
Engineering

Add each new way a change has broken production to the instructions of the AI agent that reviews every pull request.

The Pragmatic Engineer · How software engineering is changing: a tale of two eras · 4 Nov 2026
Engineering

Ask the AI agent to test its own change locally, by starting the servers, sending requests and trying edge cases, before a person reviews it.

seangoedecke.com · Getting good at AI takes time and practice · 29 Oct 2026
Engineering

Have engineers build the internal coding tools, automated pull request review and deployment automation that let designers ship their own changes.

X · 500+ PRs merged by designers! · 17 Oct 2026
Engineering

Check regularly whether reviewers still examine AI agent output or only approve it.

Thoughtworks · Decision models and the possibility of NoOps · 13 Oct 2026
Engineering

Report how long pull requests wait for review and how many are reworked afterwards, next to the amount of code produced.

LeadDev · Faster code isn’t faster delivery · 7 Oct 2026
Engineering

Require every code change written by an AI agent to pass automated reviews, including security reviews, and fixed rules that block edits to restricted code before it can merge.

All Things Distributed · AWS Emulator and the engineer's job when agents write the code · 12 Nov 2026
Engineering

Give an AI agent its own chat channel and access to logs so that it watches the health of one system component.

Anthropic Engineering · Redesigning source control at Anthropic · 14 Oct 2026
Engineering

Have an AI agent attach the relevant code, user activity, active experiments, recent pull requests and reproduction steps to each bug report before an engineer picks it up.

Duolingo Blog · We tried “Claude, fix this bug”—here’s what actually worked · 5 Nov 2026
Engineering

Write personal and professional opinions in team communication by hand, and keep AI agents to drafting other prose and code.

LeadDev · 5 shifts I'm watching as LeadDev's new Editor at Large · 16 Oct 2026
Engineering

Read the code an AI model writes before it ships.

seangoedecke.com · Shipping is the foundation · 5 Oct 2026
Engineering

Before AI agents write a feature's code, have senior engineers write the tests, interface contracts and architecture checks that the agents' code must pass in the delivery pipeline.

martinfowler.com (Rachel's Ramblings) · Why are good developers so wary of agentic development? · 30 Oct 2026
Engineering

Cap how many topics each engineer and each squad may have open at once, and hold the cap when stakeholders or engineers push to start more AI-assisted work.

Codacy · How AI Is Changing the Engineering Manager Role: More Context, More Capacity, and the New Job of Protecting Focus · 25 Sep 2026
Engineering

Set stricter automated checks for test coverage and code complexity on AI-written code than on hand-written code, and run them on every change before merging.

Codacy · How AI Is Changing the Engineering Manager Role: More Context, More Capacity, and the New Job of Protecting Focus · 25 Sep 2026
Engineering

Before an AI agent reads logs, experiments or pull requests, filter them with simple rules so it examines only the likely causes.

Duolingo Blog · We tried “Claude, fix this bug”—here’s what actually worked · 5 Nov 2026
Engineering

Use the most expensive AI models for large exploratory tasks that produce a plan, and give the plan to smaller AI models to carry out.

LeadDev · Cheap AI tokens are running out · 30 Oct 2026
Engineering

Put the engineers who build a service on call for it, including the most senior and best-paid engineers.

The Pragmatic Engineer · The revised rules of engineering management with Will Larson · 22 Oct 2026
Engineering

Build an internal tool that lets non-engineers make apps from company data and the company's own design library.

The Pragmatic Engineer · The Pulse: new trend of building “internal vibe-coding apps” at tech companies · 19 Oct 2026
Engineering

Have AI agents write the tests as well as the code, and keep spending about as much effort on tests as on the code itself.

The Pragmatic Engineer · The state of the tech industry in 2026 · 8 Oct 2026
Engineering

Credit every commit an AI agent makes to both the agent and the person who asked for the work.

LukeW · Crazy Agent Scale: Companies All-in on AI · 21 Nov 2026
Engineering

Before an engineer reviews code that a designer made with AI, run automated checks that it uses the design system's values, imports the real components and meets accessibility rules.

Slot Machine · Design Engineering Has a Handoff Problem · 18 Nov 2026
Engineering

Agree written guardrails with designers so that they can ship small layout, spacing and colour fixes to the production code themselves.

Bootcamp · From Mockups to Merged PRs: An AI-Native Designer’s Playbook · 2 Nov 2026
Engineering

Give each engineering manager responsibility for some of the team's technical decisions, instead of leaving every technical choice to the engineers they manage.

The Pragmatic Engineer · The revised rules of engineering management with Will Larson · 22 Oct 2026
Engineering

Have an automated pipeline propose and test a fix for each flagged security vulnerability, so an engineer starts from the fix and decides whether it is safe to ship.

Inside Atlassian · 3 lessons from rebuilding Atlassian’s product development lifecycle for people and agents · 13 Nov 2026
Engineering

Add an automated check that rejects code from an AI agent when it does not use the design system's components and design tokens.

Design Systems Collective · The days of recreating your product in Figma are numbered (and that’s a good thing) · 31 Oct 2026
Engineering

Send predictable operations requests to an AI agent first, and route them to a person when the agent's confidence score is low.

Thoughtworks · Decision models and the possibility of NoOps · 13 Oct 2026
Engineering

Have AI agents write end-to-end tests first and correct each one against the real system before they build the code that must pass it.

All Things Distributed · AWS Emulator and the engineer's job when agents write the code · 12 Nov 2026
Engineering

For each stage of software delivery, write down the inputs, the jobs for people and AI agents, the output and the criteria for success, and improve the stages separately.

Microsoft Command Line · The new primitives of agentic software delivery · 31 Oct 2026
Engineering

Before building on an open standard for AI agents, check that it is stable, governed by a neutral body and open to contributions.

LeadDev · Software developers are becoming agent builders · 10 Oct 2026
Engineering

Set each senior engineer a coaching goal to review and answer for every AI-assisted change they ship, as fully as for code they write by hand.

LeadDev · Long live the engineering manager · 20 Oct 2026
Engineering

When an engineer reviews a code change written by an AI agent, have them check what the change is meant to do instead of every line. Reject the change if it breaks a core rule of the system.

Anthropic Engineering · Redesigning source control at Anthropic · 14 Oct 2026
Engineering

Compare every proposed ticket, design document or delegated task against the cost of making the change directly.

seangoedecke.com · Shipping is the foundation · 5 Oct 2026
Engineering

When an engineering manager reports on AI enablement, show how the team's delivery has changed since adoption, not how many engineers use the tools.

Accidentally in Code · Judgement > Output · 21 Oct 2026
Engineering

Give employees who build their own software with AI one shared platform where IT and security teams can see which applications reach which tools and data.

LeadDev · Everyone’s a software engineer now · 17 Nov 2026

Agent ownership

A person or team answers for what an AI agent does, and decides its instructions, its default settings and when it must hand a decision to a person.

47 plays · Topic pageClearest first, as Jev scored them
CHECKEDJEV22 NOV 2026
Engineering

Count an AI agent's incident task as finished only when a separate check shows the problem has gone, such as the alarm clearing, not when the agent's action reports success.

LeadDev · Your agent loop is not a production system · 28 Sep 2026
Engineering

Enforce an AI agent's limits with fixed rules in code that decide which tools it may call, and leave every action that moves customer money to a person.

Latent Space · Agents Prompt People: Using Agents for Long-Horizon Financial Work · 1 Nov 2026
Engineering

Before an AI agent goes into production, write down the one job it is for and test that it declines requests outside that job.

Microsoft Inside Track · Scaling AI responsibly at Microsoft: Lessons from our Customer Zero journey · 24 Oct 2026
Engineering

Limit an AI agent's database changes to adding tables and columns, and leave deletions and renames out until the data can be restored.

LayerX Engineering Blog · Letting an AI agent design tables on a platform whose data structure changes with each business task · 5 Nov 2026
Engineering

Keep AI agent instructions as version-controlled files so that any change can be reviewed and rolled back.

Atlassian blog · How two companies made agentic AI the core of modern enterprise workflows · 17 Nov 2026
Engineering

Start using AI agents in one low-risk process that the rest of development does not depend on, and name one person in that team to set the agents up.

arXiv · Developing a Roadmap to an AI-first Organization: A Case Study in Embedded Software Development · 25 Sep 2026
Engineering

Ask an AI agent to check metrics and logs before and after each deploy or feature-flag change, and to keep checking at set intervals for the rest of the day.

seangoedecke.com · You can just ask AI to do things · 30 Oct 2026
Engineering

Schedule an AI agent to read overnight messages each morning, flag the questions and problems that need a reply, and look up the relevant logs before the workday starts.

seangoedecke.com · You can just ask AI to do things · 30 Oct 2026
Engineering

Save each AI agent run's progress, conversation and files to storage outside the cloud machine it runs on, so a new machine can resume the work after a failure.

PostHog · We built our own cloud agents runtime. Here's what we learned · 10 Oct 2026
Engineering

Track rework rate, code review wait time and pull request size as AI agents take on more of the coding.

LeadDev · 3 agentic AI questions engineering leaders must answer · 13 Nov 2026
Engineering

Restrict the network addresses an AI agent's machine can reach to an approved list the machine enforces, instead of relying only on an AI model that screens the agent's inputs.

PostHog · We built our own cloud agents runtime. Here's what we learned · 10 Oct 2026
Engineering

Keep a register with one row per AI agent that names its owner and a backup, its permissions and what people do if it is switched off.

NerdHeadz · Nobody Needs Your AI Agent (Until Everyone Does): A Practical Guide to Agent Sprawl · 20 Nov 2026
Engineering

Hold a monthly review that retires AI agents unused for 60 days and revokes credentials that no running agent needs.

NerdHeadz · Nobody Needs Your AI Agent (Until Everyone Does): A Practical Guide to Agent Sprawl · 20 Nov 2026
Engineering

Collect reviewers' feedback and records of each AI agent run, and use them to propose changes to the software that runs the agents.

Latent Space · Agents Prompt People: Using Agents for Long-Horizon Financial Work · 1 Nov 2026
Engineering

After every release, run a fixed set of real questions against the AI agent and check which tool it chose as well as its answers.

Snowflake Builders Blog · Agents Are Code Too: What I Learned Shipping a Cortex Agent Through Azure DevOps · 12 Nov 2026
Engineering

Count an AI agent's task as done only when automated tests and visual checks pass, and attach the results to the pull request for the engineer who approves it.

Engineering at Meta · How Meta Built Forum With AI-Native Engineering · 21 Oct 2026
Engineering

Build a shutdown control for each production AI agent that the agent cannot change, which ends the agent, revokes its credentials and keeps its logs intact.

Thoughtworks · Real-world lessons in agentic authority and overreach · 9 Oct 2026
Engineering

When AI agents repair their own failing work, cap the repair attempts and the spending, and hand the problem to an engineer once either limit is reached.

Salesforce Engineering · Engineering Multi-Agent AI Teams That Build and Test Themselves · 28 Sep 2026
Engineering

Run a nightly AI agent that reviews the previous day's agent sessions and opens pull requests to improve the agents' instructions.

Ubie Tech Blog (Zenn) · Beyond the software factory: Ubie's in-house cloud agent platform for making all company work AI-native · 13 Nov 2026
Engineering

Require a named person's approval before any AI agent merges a pull request or takes an irreversible action.

The New Stack · Moving from AI adoption to an AI operating model: How Docusign engineering went beyond tool rollout · 9 Nov 2026
Engineering

Before opening a pull request, have the coding agent render each new screen in the mobile emulator, and compare the screenshot with the approved design.

Modus Create · Putting agentic software development into practice: Lessons from a real-world project · 6 Nov 2026
Engineering

Before shipping an alert threshold for an AI agent's work, replay it against past data to see how often it would have fired.

Snowflake Builders Blog · Your Agent Ran Overnight. Here Is What It Got Wrong. · 23 Oct 2026
Engineering

Make every shared AI agent visible to all teams with its usage count, so that teams copy and extend a working agent before building their own.

The New Stack · Moving from AI adoption to an AI operating model: How Docusign engineering went beyond tool rollout · 9 Nov 2026
Engineering

Map every active AI agent workflow and assign one named person, not a team, to answer for each workflow's output before it runs again.

InformationWeek · When AI agents collaborate, who owns the result? · 3 Nov 2026
Engineering

Name one person as the owner of each production AI agent, with a written list of actions it may never take and a named approver for changes.

Thoughtworks · Real-world lessons in agentic authority and overreach · 9 Oct 2026
Engineering

Give each AI coding agent one whole feature, from interface to storage, instead of one layer such as the frontend, so that agents seldom need to brief each other.

Medium · Durable Agentic Teams: Maximizing Your Software Development Workflow · 29 Sep 2026
Engineering

Keep a register of each AI agent's decision rights, with an escalation route and an audit trail.

Thoughtworks · Leadership in the age of AI agents · 10 Oct 2026
Engineering

When a team hands one low-risk development step to an AI agent, measure how review and testing time change in the steps before and after it, not only the time saved on that step.

arXiv · Developing a Roadmap to an AI-first Organization: A Case Study in Embedded Software Development · 25 Sep 2026
Engineering

At each handoff between AI agents, record what the agent was trying to do, what data it had, how confident it was and what action it took.

InformationWeek · When AI agents collaborate, who owns the result? · 3 Nov 2026
Engineering

When an AI agent repeats the same mistake more than twice, write the correction into the project's permanent rules for the agent.

Engineering at Meta · How Meta Built Forum With AI-Native Engineering · 21 Oct 2026
Engineering

Copy build and test run data into a database that AI agents can query, instead of having agents call the code host for every question.

PostHog · Engineers are shipping faster, so we made our CI self-driving to keep up · 29 Oct 2026
Engineering

Turn the team's rules on what AI agents may change without a person's approval into automated checks in the deployment pipeline that block any change breaking them.

LeadDev · 3 agentic AI questions engineering leaders must answer · 13 Nov 2026
Engineering

Keep an AI agent's instructions, tool descriptions, data definitions and access rights in one repository and release them together through the usual approval gates.

Snowflake Builders Blog · Agents Are Code Too: What I Learned Shipping a Cortex Agent Through Azure DevOps · 12 Nov 2026
Engineering

Before an unattended AI coding agent pushes a fix to a pull request, have a separate script check whether the push is allowed, and in approval mode wait for a person to reply with a single-use token.

Snowflake Builders Blog · Your Agent Ran Overnight. Here Is What It Got Wrong. · 23 Oct 2026
Engineering

Run an AI agent in a shared Slack channel under a service account that sees only what every channel member may see, instead of the requester's own access.

Ubie Tech Blog (Zenn) · Beyond the software factory: Ubie's in-house cloud agent platform for making all company work AI-native · 13 Nov 2026
Engineering

Give the AI agents that staff build a shared company workspace with access rules, version history and a record of what each agent did, instead of banning them.

a16z · Let Chaos Reign, and Then Rein In Chaos · 27 Oct 2026
Engineering

Have AI agents that check other agents' work answer pass, warn or fail in a fixed format, and compare several checkers' answers instead of trusting a single one.

Salesforce Engineering · Engineering Multi-Agent AI Teams That Build and Test Themselves · 28 Sep 2026
Engineering

Show each question an AI agent raises with the decision first, then the facts, then the options, so a person can answer by number.

Rescue Now Tech Blog · Leaving task management to AI agents too, until the only work left was answering their confirmation requests · 8 Nov 2026
Engineering

Store with each file version the AI agent conversation that produced it, and let a person undo one conversation by restoring only the files it changed, saving a renamed copy where someone has edited a file since.

Dropbox.Tech · How two engineer side projects became the agent taking action inside Dropbox · 6 Nov 2026
Engineering

Cap how many issues a new AI agent task may create in one run while the team checks its output.

Linear · Five principles for working with Loops · 14 Oct 2026
Engineering

When one AI agent hands tasks to other agents, make it wait for an engineer's approval before anything colleagues will see, such as a pull request sent for review.

Rescue Now Tech Blog · Leaving task management to AI agents too, until the only work left was answering their confirmation requests · 8 Nov 2026
Engineering

Keep credentials out of an AI agent's environment and let a proxy add short-lived credentials only for approved destinations.

Shopify Engineering · Cyber Chaos Monkey: Staying ahead of security threats with AI agents · 13 Nov 2026
Engineering

Start each new AI agent with access to nothing, and grant each resource only when its work requires it.

Microsoft Inside Track · Launching the personal agent era: How we’re rolling out Autopilot at Microsoft · 20 Nov 2026
Engineering

Give every always-on AI agent its own identity, a named human sponsor and a stated business purpose.

Microsoft Inside Track · Launching the personal agent era: How we’re rolling out Autopilot at Microsoft · 20 Nov 2026
Engineering

Start a recurring AI agent task with a narrow scope that only suggests work, and let it assign work only after its suggestions prove reliable.

Linear · Five principles for working with Loops · 14 Oct 2026
Engineering

Run a second AI agent on a different AI model to read every action of an agent on live systems and stop it when it breaks agreed rules.

Shopify Engineering · Cyber Chaos Monkey: Staying ahead of security threats with AI agents · 13 Nov 2026
Engineering

Let AI agents work on the build pipeline only when one of a few fixed queries flags a problem, such as a flaky check, a failing main branch or a slower workflow.

PostHog · Engineers are shipping faster, so we made our CI self-driving to keep up · 29 Oct 2026

Agent context files

The files an AI agent reads before it works, such as AGENTS.md, CLAUDE.md, design-system rules and other written instructions, and the person or team who keeps them current.

32 plays · Topic pageClearest first, as Jev scored them
CHECKED BY JEV · CHECKED BY JEV ·22.11.26
Engineering

Have an AI model draft a glossary of company terms from internal documents and chat logs, then have experts in the business correct it before AI agents use it.

martinfowler.com · Generated Glossary · 16 Nov 2026
Engineering

Test candidate AI models on the team's own languages and frameworks before choosing a company-wide default.

LeadDev · AI coding rules matter more than your model · 19 Nov 2026
Engineering

Add a table to the AI agent's project instructions saying where each kind of note belongs, by how long it stays true and who keeps it current.

PostHog · How I use skills for everything · 21 Oct 2026
Engineering

Test AI agents on the same tasks with and without their added instruction files, because extra context can confuse an agent and make its results worse.

Design Systems Collective · “You don’t fix the output, you fix the system” — Cristian Morales Achiardi on agentic design systems · 29 Sep 2026
Engineering

Write down how senior engineers investigate a problem, including their checks, trusted signals and order of steps, and turn it into instructions an AI agent follows.

LeadDev · Meta turned engineers’ judgment into agent skills · 6 Oct 2026
Engineering

Give AI coding agents a small local tool that answers narrow questions about the design system, such as which properties a component takes, instead of loading every component's documentation on every turn.

Headway · Your Design System MCP Shouldn't Be Costing You Tokens · 13 Nov 2026
Engineering

When a bug reaches users, fix the automated check that should have caught it, for example by running screenshot tests at phone width, before closing the bug.

Accidentally in Code · The role of engineers in the software factory · 4 Nov 2026
Engineering

Give a coding AI agent the same build tasks several times with the design system's instructions and documents and several times without them, then score every result with pass-or-fail checks.

Design Systems Collective · Four feedback loops for an AI-native design system · 26 Oct 2026
Engineering

Have the AI agent update a shared handover file and change log at the end of every working session.

PostHog · How I use skills for everything · 21 Oct 2026
Engineering

Add an automated check to the build that renders every interface component at phone width, records the layout overflows that exist today as a baseline that may only shrink, and fails the build on any new overflow.

Accidentally in Code · The role of engineers in the software factory · 4 Nov 2026
Engineering

Put a short index file in each folder of an AI agent's context files that says what the folder holds and when the agent should read it.

Nielsen Norman Group · Context Libraries: How AI Power Users Manage Context Engineering · 11 Oct 2026
Engineering

Search AI-written tests for known worthless patterns, such as tests that expect a server error, and review a sample of files by hand.

LITALICO Engineers (Zenn) · How test coverage was raised from 23% to 84% with AI · 11 Nov 2026
Engineering

Let AI coding agents look up current colour values and component options directly from the design system, instead of from a summary file that goes out of date.

Supernova · DESIGN.md for Design Systems: What Fits in the File · 7 Nov 2026
Engineering

Publish the design system's components as installable packages that an AI coding agent can read, not only as written guidelines.

DEV Community · Does a design system change what a coding agent writes? · 14 Oct 2026
Engineering

Write a file of security rules for the team's AI coding tools in each programming language the team uses, and test the rules again whenever the team changes AI model.

LeadDev · AI coding rules matter more than your model · 19 Nov 2026
Engineering

Store the instructions every AI agent task reads, the files for one task and incoming streams such as email in three separate folders.

Nielsen Norman Group · The New Big Ball of Mud: Why Agentic AI Systems Turn Fragile · 3 Oct 2026
Engineering

When users in different roles see the same data, keep every rule about who may see what in one central permissions table in the code, so that granting a permission is a one-line change.

Accidentally in Code · Good Architecture Is a Function of Time · 29 Sep 2026
Engineering

Add a lint rule to the front-end build that fails when component code uses a raw hex colour instead of a design-system colour token, so that AI coding agents correct it.

Medium · Your Design System Has a New User, and It Can’t See Your Figma File · 29 Oct 2026
Engineering

Ask the AI agent to report the problems it met after every run, and treat that report as a list of fixes.

PostHog · WTF is context engineering? (with real examples) · 10 Oct 2026
Engineering

After each AI agent run, save what the agent had to learn, the decisions it made alone and the engineers' review comments, and update the reusable prompts from them.

Stripe Dot Dev Blog · Stripe’s Payment Method Factory: Orchestrating agents for repeated, custom integrations · 3 Oct 2026
Engineering

Compare the tokens an AI agent spends building one screen when it reads one long design rules file in full and when it fetches only the rules it needs.

Headway · Your Design System MCP Shouldn't Be Costing You Tokens · 13 Nov 2026
Engineering

In the product repository's AI agent instruction file, require the agent to import components from the design system package and use its design tokens for colour and spacing.

Supernova · What Your AGENTS.md Should Say About Your Design System · 20 Nov 2026
Engineering

Write the steps an experienced engineer follows to write a test, and the team's testing trade-offs, into a skill the AI agent reads before writing tests.

LITALICO Engineers (Zenn) · How test coverage was raised from 23% to 84% with AI · 11 Nov 2026
Engineering

Use one AI agent session to write the implementation prompt from the requirements, and have a person review it before a second session writes the code.

Qiita · How development started to flow after separating the Claude that writes prompts from the Claude that implements · 8 Nov 2026
Engineering

Score an AI model only on hand-settled cases that the examples in its prompt never used, so the score measures understanding rather than memory.

Design Systems Collective · I tested 12 small AI models on 2,500 design-system questions. The model wasn’t what mattered. · 27 Oct 2026
Engineering

Generate from the code a file listing every design system component with its properties, allowed values and defaults, and connect AI coding agents to it so they build prototypes only from real components.

UX Collective · AI Prototyping inside your design system. A hands-on guide for design & product teams. · 13 Nov 2026
Engineering

Distribute shared AI agent instructions and skills to every engineer through one plugin that updates itself, instead of asking each engineer to install each skill.

Salesforce Engineering · How Salesforce Scaled Engineering AI Adoption Without Stifling Innovation · 11 Nov 2026
Engineering

Split agent work into steps that each map to one pull request, and run the coordination between steps in ordinary code rather than in an AI agent.

Stripe Dot Dev Blog · Stripe’s Payment Method Factory: Orchestrating agents for repeated, custom integrations · 3 Oct 2026
Engineering

Make the team that adds a skill to the engineers' shared AI coding plugin deliver and support that skill, with architects setting the quality checks it must pass.

Salesforce Engineering · How Salesforce Scaled Engineering AI Adoption Without Stifling Innovation · 11 Nov 2026
Engineering

When AI agents write much of the code, invest in shared structure for the parts that change most, such as a design system or generated database queries.

Accidentally in Code · Good Architecture Is a Function of Time · 29 Sep 2026

Decision records

A written record of what was decided and why, kept with the work so that someone who did not build it can review it.

20 plays · Topic pageClearest first, as Jev scored them
CHECKEDJEV22 NOV 2026
Engineering

Have a second AI model from another maker review the code that the first AI model writes.

Latent Space · AI for the Fortune 2000: How Tenex Builds an AI-Native SDLC — Alex Lieberman & CJ Hess · 22 Oct 2026
Engineering

Record every test case left out of a release with the reason it was left out, so the choice can be reviewed after a bug.

PIVOT Tech Blog · A record of choosing which ever-growing AI-written test cases to run, using risk-based testing · 14 Nov 2026
Engineering

Keep a short list of the assumptions behind each major engineering plan, each with a signal to watch, a named person responsible and a condition that triggers review.

LeadDev · Your engineering strategy has a new failure mode · 13 Oct 2026
Engineering

Keep one file in the code repository that lists the team's settled engineering decisions for coding agents, and update it in the same commit that changes a decision.

SSD Nodes · DESIGN.md: the file after AGENTS.md · 11 Oct 2026
Engineering

Commit plans and decision records to the repository beside the code, so that an AI agent returning to a task can read why the team chose its approach.

Latent Space · AI for the Fortune 2000: How Tenex Builds an AI-Native SDLC — Alex Lieberman & CJ Hess · 22 Oct 2026
Engineering

Sort a year of code review comments into repeated rules and one-off decisions, and turn the repeated rules into checks that run on every pull request.

The New Stack · How we review 4x more AI code without reading every line · 8 Nov 2026
Engineering

Turn each written design rule into an automated check that blocks a code change breaking it, for example a colour typed by hand where a named design-system colour belongs.

ElGuerre · Why design before code when an AI agent builds the screen in minutes: sketch, mockup and prototype · 9 Oct 2026
Engineering

Ask for a two-page design brief covering the approach, the alternatives and the failure modes before AI agents change code that crosses services or alters stored data.

LeadDev · Are technical design reviews disappearing? · 28 Oct 2026
Engineering

Require every AI-written pull request to include a file listing each check the AI agent ran and each thing it could not verify, before the pull request leaves draft.

Irrational Exuberance · Self-reinforcing quality / adopting verification pattern · 31 Oct 2026
Engineering

In code review of AI-written pull requests, look first for changes that the approved plan did not ask for.

Zenn · Design review rather than code review · 3 Nov 2026
Engineering

Write the rules a coding agent must not break, each with its reason, in one file in the code repository. Test the file by giving the agent a task that touches a rule and checking whether the agent cites it.

SSD Nodes · DESIGN.md: the file after AGENTS.md · 11 Oct 2026
Engineering

Open each AI agent's pull request with a short summary a person has checked, and fold the agent's full description into a collapsed section.

Simon Willison’s Weblog · Hide your slop PR descriptions behind <details><summary> · 10 Nov 2026
Engineering

Before a release, choose test cases with a written table that rates each acceptance criterion by how badly its failure would harm users and whether the release changes it.

PIVOT Tech Blog · A record of choosing which ever-growing AI-written test cases to run, using risk-based testing · 14 Nov 2026
Engineering

In each engineering plan that counts on AI tools, name a person to reassess it if code review capacity grows more slowly than the code being written.

LeadDev · Your engineering strategy has a new failure mode · 13 Oct 2026
Engineering

Have engineers read each day's pull request files listing what AI agents checked, and open an issue for each recurring gap in what the agents could not check.

Irrational Exuberance · Self-reinforcing quality / adopting verification pattern · 31 Oct 2026
Engineering

Skip the two-page design review for code changes that stay inside one service, leave stored data alone and are easy to undo, even when an AI agent wrote them.

LeadDev · Are technical design reviews disappearing? · 28 Oct 2026
Engineering

List every decision made during an AI-written change in its pull request, with where the decision came from and who settled it.

The New Stack · How we review 4x more AI code without reading every line · 8 Nov 2026
Engineering

Review and merge a written plan for any change that contains a decision someone could disagree with, then let AI agents implement against it.

Giant Swarm · Spec-driven development: how we review plans, not pull requests · 15 Oct 2026
Engineering

Before any code is written, have a person check the AI agent's plan against the requirements, the code it will touch and the shape of data and interfaces.

Zenn · Design review rather than code review · 3 Nov 2026
Engineering

Have an AI agent check every factual claim in a plan against the real code before people approve the plan.

Giant Swarm · Spec-driven development: how we review plans, not pull requests · 15 Oct 2026

Evals before the build

Before an AI feature is built, the team writes down what a good output looks like, so that the result can be tested against it.

12 plays · Topic pageClearest first, as Jev scored them
CHECKEDJ22 NOV 2026
Engineering

Before an AI agent writes code, require a short written spec that says what the change does, how to tell it works and what it must not break.

Microsoft Command Line · Nine engineers, 1,211 pull requests in a day: Lessons from building Copilot Cowork · 5 Nov 2026
Engineering

Before trusting one AI model to grade another's output, label a set of examples by hand and measure how often the grader agrees.

Product Talk · 4 New Evals and 16 Experiment Variants to Fix 1 Customer Complaint · 17 Sep 2026
Engineering

After each change to an AI agent's instructions, rerun its tests and list every case that flipped between pass and fail, instead of trusting the total score.

Pinterest Engineering Blog · How Pinterest Improved Analytics Agent Compliance with Loop Engineering · 13 Nov 2026
Engineering

Do not count tests that an AI model wrote for existing code as verification, because they record what the code already does, errors included.

BLOG@CACM · Nobody Did TDD for 25 Years. Now the Machine Requires It · 28 Sep 2026
Engineering

Before adopting an extra step in an AI coding workflow, run the same tasks with and without it and compare correctness against time and token cost.

Thoughtworks · SpecBench: A tool to evaluate AI-enabled software development workflows · 14 Nov 2026
Engineering

Have the business owner approve 20 to 30 example requests and their expected answers before measuring an AI agent's first baseline.

Amazon Web Services · From prototype to production: Engineering agents with Agent DLC · 15 Nov 2026
ProductEngineering

Before an AI agent starts building, agree with the people who approve its changes which automatic tests every change must pass.

From 3 articles · Anthropic, GitHub Next, PikdShow all 3Hide
Anthropic · in Anthropic · 24 Sep 2026The automatic checks are agreed with the people who approve changes, and tests are added until those engineers would merge on the results alone.
GitHub Next · in The Pragmatic Engineer · 26 Sep 2026The spec for a prototype lists how the agent will verify each part of its work.
Pikd · in BLOG@CACM · 28 Sep 2026The tests that define a correct result are written first, the agent works until they pass, and people review the tests instead of the code.
Engineering

When an engineering team changes an AI agent's instructions, test each rule against two sets of real chats, one where the rule should apply and one where it should not, and keep the change only if both pass.

Pinterest Engineering Blog · How Pinterest Improved Analytics Agent Compliance with Loop Engineering · 13 Nov 2026
Engineering

Give every new automated check a test that makes it fail on purpose, and have each check report how many files it examined.

Microsoft Command Line · Nine engineers, 1,211 pull requests in a day: Lessons from building Copilot Cowork · 5 Nov 2026
Engineering

Turn failed conversations into test personas, and run them again whenever the AI model or provider changes.

Product Talk · Building an AI Voice Agent for 360 Degree Feedback: How Your360 AI Uses State Machines, Evals, and Layered Synthesis · 16 Oct 2026
Engineering

When an AI agent fails on a case that no criterion covers, add the criterion and its examples first, then change the agent to meet them.

Amazon Web Services · From prototype to production: Engineering agents with Agent DLC · 15 Nov 2026

Ground truth for review

A set of correct answers or expected results, prepared by someone other than the builder, that a reviewer uses to check work made with AI.

12 plays · Topic pageClearest first, as Jev scored them
22 NOV2026CHECKED BY JEV
Engineering

Run the full automated test suite once as a final acceptance check before accepting an AI agent's work, even when a frontier AI model passes first time.

martinfowler.com · Are Harnesses Worth the Cost? · 14 Oct 2026
Engineering

When testing an AI agent, run each test task many times and report how many tasks it passed on every attempt beside its average pass rate.

Microsoft Command Line · The check that catches agent failures can also train them · 7 Nov 2026
Engineering

Make test data files read-only to coding agents, or add a continuous integration check that flags unexpected changes to them.

martinfowler.com · Tests as data: Ensuring AI-generated code does what we want · 12 Nov 2026
Engineering

Have an AI coding agent write down the test results it expects, then have a separate tool run the tests and compare the actual results with those expectations.

GENDA (Zenn) · Blocking an AI agent's shortcuts with tools: what checking code comments showed about the roles of steering and MCP · 15 Nov 2026
Engineering

Give a separate testing AI agent only the original requirements and the running app, so it checks behaviour without sharing the coding agent's assumptions.

Stack Overflow Blog · Your coding agent shouldn't grade its own homework · 14 Nov 2026
Engineering

Audit a sample of AI-written unit tests by hand and count how many would catch a real defect.

Stack Overflow Blog · Your coding agent shouldn't grade its own homework · 14 Nov 2026
Engineering

Write customer tests as tables of inputs and expected results in plain terms, and have an AI agent write the program that runs them.

martinfowler.com · Tests as data: Ensuring AI-generated code does what we want · 12 Nov 2026
Engineering

Keep acceptance tests written by people separate from the coding agent, and instruct the agent to fix its code instead of changing expected results.

martinfowler.com · Are Harnesses Worth the Cost? · 14 Oct 2026
Engineering

Grade an AI agent by checking the records it leaves behind, not by reading its own report of what it did.

Microsoft Command Line · The check that catches agent failures can also train them · 7 Nov 2026
Engineering

Run the AI evaluations on every pull request in an isolated preview environment, and block the merge when quality scores fall below set thresholds.

Netflix TechBlog · Making Failure Diagnosis Reliable with Pensive AI · 27 Oct 2026
Engineering

When adding an AI model to a rule-based system, keep the old hand-written rules as reference answers that each new version of the model must match before it ships.

Netflix TechBlog · Making Failure Diagnosis Reliable with Pensive AI · 27 Oct 2026
Engineering

Before an AI coding agent runs each test, have it lock its predicted pass or fail result, and let it change the test later only with a written reason.

GENDA (Zenn) · Blocking an AI agent's shortcuts with tools: what checking code comments showed about the roles of steering and MCP · 15 Nov 2026

Understanding the work

A team checks on purpose that its people can still explain work that AI agents produced.

12 plays · Topic pageClearest first, as Jev scored them
CHECKED22.11.26JEV
Engineering

Have the engineer writing a test for a new feature run it once with the feature missing and confirm it fails, so that a later pass shows it measures something.

Jepsen · Jepsen: ParticleDB 1.1.4 · 2 Nov 2026
Engineering

Ask an AI agent fixing a bug to record the broken behaviour, apply the fix and repeat the same steps, so the reviewer sees the result before and after.

Shopify Engineering · We gave AI agents a mobile dev environment · 25 Oct 2026
Engineering

When reviewing a large code change, first trace one important path through the functions it touches, then read the changed lines from start to finish.

Sean Goedecke · How to read code · 8 Oct 2026
Engineering

Measure defect rates and mean time to resolution, not lines of code, to judge whether AI coding tools speed up delivery.

LeadDev · AI code causes real world production outages · 13 Oct 2026
Engineering

Check a sample of required documents, such as risk assessments, against the work they describe.

seangoedecke.com · Writing is no longer proof of work · 19 Oct 2026
Engineering

Ask the team that owns a critical service to explain its recent consequential changes without its go-to engineer or its AI tool before approving the next release.

LeadDev · Your error budgets don't know AI exists · 1 Oct 2026
Engineering

Sort the team's documents into those colleagues will actually read and those that only record a step, such as risk assessments, and spend careful writing only on the first kind.

seangoedecke.com · Writing is no longer proof of work · 19 Oct 2026
Engineering

Once AI agents write most of the code, have engineers present their finished work to the team, so that each keeps pride in the work and knows what colleagues build.

shubs.io · do we still enjoy software engineering in the age of AI? · 26 Sep 2026
Engineering

Have a low-cost AI model label every coding agent conversation by outcome, and use the failed conversations to rewrite the agent's instructions.

Shopify Engineering · We gave AI agents a mobile dev environment · 25 Oct 2026
Engineering

Write at least one simple example-based test for every feature, and run the whole suite before each release.

Jepsen · Jepsen: ParticleDB 1.1.4 · 2 Nov 2026
Engineering

Give the team that owns a critical service the authority to delay a release when it cannot explain a change, before any reliability target is breached.

LeadDev · Your error budgets don't know AI exists · 1 Oct 2026
Engineering

Read an AI reviewer's comments as carefully as the code, and still read the code itself.

Sean Goedecke · How to read code · 8 Oct 2026

Work cut to size

Before an AI agent starts, the work is split into pieces small enough for a person to review one at a time.

12 plays · Topic pageClearest first, as Jev scored them
CHECKEDJEV22.11.26
Engineering

When a team plans its work, set aside review time for pull requests produced with AI coding agents and name who has authority to approve them.

The Hybrid Leader · How Many AI Agents Can You Actually Manage? · 22 Nov 2026
Engineering

Run automated checks, such as behaviour tests and independent code reviews, before a person approves agent-built work, so that each piece reaches the person already checked.

Shopify Engineering · Helix: The internal tool powering our Shopify app's native migration · 23 Sep 2026
Engineering

Have the AI agent test each database query change on a read-only copy of the database before it opens a pull request.

Sentry Blog · Finding backend performance wins with a coding agent · 7 Nov 2026
Engineering

Start each task given to a coding agent with a written plan agreed by the engineer and the agent, and have the agent record its test results for review.

Dropbox.Tech · How AI-assisted development helped us build the new Dropbox experience · 6 Nov 2026
Engineering

Split long-deferred cleanup work into small defined tasks that coding agents complete through the same test and review process.

Dropbox.Tech · How AI-assisted development helped us build the new Dropbox experience · 6 Nov 2026
Engineering

At the end of each day, turn the day's code review comments into rules in the files that AI agents read before they work.

Shopify Engineering · How we rebuilt the Shopify app in 12 weeks · 11 Nov 2026
Engineering

Before AI agents build a large feature, have a person review its written specification, and add a feature flag and a rollout checklist.

Accidentally in Code · Seeing the bottlenecks in the software factory · 18 Nov 2026
Engineering

Before starting another AI coding agent, count the agent tasks that are finished or paused and still need the developer's review or decision.

The Hybrid Leader · How Many AI Agents Can You Actually Manage? · 22 Nov 2026
Engineering

Have an AI agent review each agent-written pull request against the team's architecture rules before a person reviews it.

Shopify Engineering · How we rebuilt the Shopify app in 12 weeks · 11 Nov 2026
Engineering

Have the AI agent read the code and its change history and state what might break before it changes anything.

Sentry Blog · Finding backend performance wins with a coding agent · 7 Nov 2026
Engineering

On a regular schedule, clear the column of tasks marked ready on the team's board, to find work that has stalled close to completion.

Accidentally in Code · Seeing the bottlenecks in the software factory · 18 Nov 2026
Engineering

Cut each agent-built feature into thin slices that each work end to end, and review the agent's implementation plan as a group before any code is written.

SEP · So, AI Killed Your Code Review Process. Now What? · 23 Sep 2026

After the prototype

AI makes a working prototype quick, so the effort moves to choosing a direction and getting from prototype to production with a reliable product.

6 plays · Topic pageClearest first, as Jev scored them
CHECKED BY JEV · WORKING SURFACE ·22 NOV2026
Engineering

To test a code base's quality, give an AI coding agent a small change to make in it and note where it wastes effort or gets confused.

Thoughtworks Insights · Cheaper code won't kill software development, but it’s already reshaping it · 20 Oct 2026
Engineering

Before an AI agent runs with broad permissions, restrict the files and systems it can change, limit its network access and run endpoint protection on its machine.

Thoughtworks · Quality is speed: Where engineering discipline moves in the AI era · 8 Oct 2026
Engineering

Write a document that links each design-system colour, spacing value and component to the production code it matches, give it to the AI agent that builds prototypes, and correct it when engineers reject the agent's pull requests.

How I AI, Lenny's Newsletter · Why Reddit built its own AI prototyping tool | Blair Metcalf · 27 Oct 2026
Engineering

Throw away a prototype built to learn, and design its data model again before any of it goes into the product.

Product Talk · Delivery Isn't Free, All Things Product Podcast · 17 Sep 2026
Engineering

When an AI agent's written instructions for a task give different results from run to run, have the agent run a fixed script for the critical step, such as extracting icon names from a Figma file.

Into Design Systems + AI · How Zalando built its own AI prototyping environment on top of its design system · 13 Nov 2026
Engineering

Let a group of no more than five people build a working version end to end behind a staff-only feature flag before reconciling the details with other teams.

Irrational Exuberance · The Explorer Pattern for product invention and development. · 22 Nov 2026

Training juniors

How people new to a craft build judgement when AI tools do the work that used to teach it.

6 plays · Topic pageClearest first, as Jev scored them
WORKING SURFACECHECKEDBY JEV · 22 NOV 2026
Engineering

Never assign a junior engineer a project alone, and have a senior engineer review the junior's pull requests, including code written with AI.

LeadDev · AI-coding tools could be breaking the junior engineer pipeline · 6 Oct 2026
Engineering

Once a week, have each junior engineer explain aloud to a senior engineer the decisions in one merged pull request.

LeadDev · Your junior engineers are getting faster but are they getting better? · 9 Oct 2026
Engineering

Mark some onboarding tickets as learning tasks that a junior engineer attempts without AI tools first, then explains in writing how the AI's approach differed.

LeadDev · Your junior engineers are getting faster but are they getting better? · 9 Oct 2026
Engineering

Run incident reviews that ask what the team believed that turned out to be untrue, so that junior engineers practise judging their own assumptions.

LeadDev · AI makes critical thinking harder to build · 29 Sep 2026
Engineering

Teach junior engineers to review what an AI tool produces and to explain why some of its choices may cause problems.

LeadDev · AI-coding tools could be breaking the junior engineer pipeline · 6 Oct 2026
Engineering

Write critical thinking into career frameworks as specific behaviours, such as validating assumptions before implementation, so that managers can see a junior engineer's growth that shipped tickets no longer show.

LeadDev · AI makes critical thinking harder to build · 29 Sep 2026

Interfaces for agents

Product pages and data designed so that an AI agent, not only a person, can use them.

2 plays · Topic pageClearest first, as Jev scored them
CHECKEDJEV22 NOV 2026

Other plays

Plays from links that are not filed under a topic.

19 playsClearest first, as Jev scored them
CHECKED BY JEV · CHECKED BY JEV ·22.11.26
Engineering

Group related issues into one batch before giving them to an AI coding agent, so that one round of coding and one test run closes them all.

Zenn · For the days CI gets jammed: a record of verification improvements that tripled closed issues in fully automated AI agent development · 22 Nov 2026
Engineering

Compare user stories closed before and after AI adoption, not only pull request counts, to see whether more code means more delivered work.

The Pragmatic Engineer · What’s the ROI of software engineers’ spending on AI? · 21 Oct 2026
Engineering

Require a test coverage check to pass before any pull request merges, because an AI coding agent may ignore a testing rule written in its instruction file.

Accidentally in Code · Is the software factory working? · 7 Oct 2026
Engineering

Record a baseline of standard delivery metrics before rolling out an AI coding tool, so its effect on speed and quality can be measured.

LeadDev · AI adoption has left strategy behind · 14 Nov 2026
Engineering

When AI agents build the same feature separately for iOS and Android, make both versions pass one shared test suite for the business logic before either ships.

The Pragmatic Engineer · Why has Shopify dropped React Native? · 30 Sep 2026
Engineering

Each month, run a script that sorts the team's pull requests into features, automated checks and same-day emergency fixes, and compare the three counts side by side.

Accidentally in Code · Is the software factory working? · 7 Oct 2026
Engineering

Offer each new AI model to staff on release day as an experiment with a capped budget, and keep it only if benchmarks, user reports and cost data support it.

Databricks · How Databricks rolls out frontier models to 12,000 employees on Day 1 · 28 Sep 2026
Engineering

When comparing AI code review tools, report each tool's share of comments that led to a code change next to its number of comments per pull request.

Zenn · The more elaborate the code review skill, the less often its comments were fixed: results of six months of checking · 16 Nov 2026
Engineering

Compare a new AI model's cost per session on the same group of early users before and after, because early users are heavier users of AI than most staff.

Databricks · How Databricks rolls out frontier models to 12,000 employees on Day 1 · 28 Sep 2026
Engineering

Have the same developer build each mobile feature on both iOS and Android with AI coding agents, instead of keeping separate iOS and Android teams.

The Pragmatic Engineer · Why has Shopify dropped React Native? · 30 Sep 2026
Engineering

Before an engineering leader prices a hard but bounded task as senior engineers' time, have an AI agent attempt it first, with a person setting the design and review standard.

LeadDev · Stop blaming AI tools for flat velocity · 18 Nov 2026
Engineering

Before buying more AI coding tools, compare lead time on similar pieces of work from before and after adoption, including tickets still unfinished.

LeadDev · Your engineers got twice as fast. Your lead time only moved 4% · 16 Nov 2026
Engineering

Before an AI model grades an AI agent's output, run a free check in ordinary code first, such as a rule that flags a list with too many items.

Product Talk · Generating Opportunity Solution Trees with AI: How Vistaly Rebuilt Its Product Around Interview Synthesis, Evals, and Repair Loops · 5 Oct 2026
Engineering

Run an AI agent on each pull request that flags oversized changes and verbose code before a person reviews it.

The Pragmatic Engineer · What’s the ROI of software engineers’ spending on AI? · 21 Oct 2026
Engineering

Have AI agents merge their branches into one integration branch and run the full test suite once on that frozen branch before a single pull request to release.

Zenn · For the days CI gets jammed: a record of verification improvements that tripled closed issues in fully automated AI agent development · 22 Nov 2026
Engineering

Involve engineers in writing the rules for how AI is used, so the people who review AI output also set the standard it must meet.

LeadDev · AI adoption has left strategy behind · 14 Nov 2026
Engineering

Have engineering managers report lead time split into waiting time, active work and rework next to AI adoption figures, with a specific reason recorded for every wait.

LeadDev · Your engineers got twice as fast. Your lead time only moved 4% · 16 Nov 2026
Engineering

Define once what counts as an AI review comment being fixed, such as the code changing afterwards, and recount every review tool against that definition each month.

Zenn · The more elaborate the code review skill, the less often its comments were fixed: results of six months of checking · 16 Nov 2026
Engineering

Read AI credit spending next to merged pull requests and ticket closure rates, and do not treat spending alone as a measure of delivery.

LeadDev · Stop blaming AI tools for flat velocity · 18 Nov 2026