Code Review When AI Wrote the Diff: What to Check, What to Automate, What to Ask

Reviewing AI-generated code means reviewing for plausibility, not just for mistakes. The diff will look clean, consistent, and confident, and it will still call a method that does not exist in your library version, swallow the exception that mattered, or ship a test edited until it passed. Review in a fixed order, intent then tests then code, automate everything mechanical, and require a pull request description that says how the change was verified.

A developer I work with opened a pull request for a new endpoint that returns the status of an outgoing payment. Two hundred lines, tidy, well named, with tests. The reviewer approved it in six minutes. In system integration testing it returned ACSC for payments that had been rejected, because the mapping from the internal status to the ISO 20022 payment status code had a default branch: anything unrecognised became settled. The assistant had written a reasonable-looking default. Nobody had asked whether a default should exist at all.

The code was not badly written. It was plausibly written, and plausibility is the thing a tired reviewer approves. Code review has to change shape when the author of the first draft is a model, and this article is the shape I now use. Review is also where the guardrails on what an assistant is allowed to touch become real, which is the part of AI Agents at Work for Analysts most developers ask me about.

Who is accountable for an AI-generated pull request?

The author. Always. That is not a moral position, it is an operational one: in an incident at two in the morning, someone has to explain why the code does what it does, and “the assistant wrote it” does not restore service.

Two practical consequences. The author must be able to explain every line of the diff before requesting review; if they cannot, the pull request is not ready. And the reviewer is entitled to ask “why is this here?” about any line and get a real answer. Teams that relax either rule accumulate code nobody on the team understands, which is a slower and more expensive kind of outage.

What failure modes does AI-generated code have?

Different ones from tired humans. These are the eight I see most, roughly in order of frequency:

Failure mode	What it looks like	How to catch it
Invented API	A method or option that does not exist in your pinned library version	Compile, type-check, and run it; check the version’s docs
Lookalike dependency	A new package whose name is almost right, or does not exist yet (slopsquatting risk)	Verify every new dependency in the registry; block unknown packages in CI
Swallowed error	except Exception: log.warning(...) where the caller needed to know	Read every catch block; ask what the caller sees
Silent default	A fallback branch that turns unknown input into a valid-looking output	Ask whether a default should exist at all
Duplicated logic	A new helper that re-implements one already in the codebase	Search for the concept, not the name
Scope creep	Renames, reformatting, and “improvements” outside the ticket	Diff size versus ticket size; split the PR
Outdated pattern	A deprecated API or an old idiom from training data	Lint rules for deprecations; team conventions doc
Edited test	An existing assertion changed so the build goes green	Review test diffs first; require a requirement change to justify it

The last row is the one to be ruthless about. A test that failed and was changed to pass without a requirement change is not a fix, it is the removal of evidence. It gets a blocking comment every time, whoever or whatever made the change.

In what order should you review a pull request?

Intent, tests, boundaries, failure paths, security, fit. The order matters because each step can end the review early, and because the most expensive thing a reviewer can approve is a well-written change that solves the wrong problem.

1. Intent. Read the ticket, then the PR description, then the list of changed files. Does the change do what the ticket needs and nothing more? If the file list surprises you, stop and ask before reading code.
2. Tests. Read the test diff before the code diff. Are new tests named after rules? Where did the expected values come from? Did any existing test change? The detail is in Developer Testing in the AI Era.
3. Boundaries. Every place data enters or leaves: API handlers, message consumers, database writes, external calls. Is input validated? Are the contract and the HTTP status codes right? Is a breaking change hiding in a renamed field?
4. Failure paths. Every catch block, timeout, retry, and default. What does the caller see when this fails? Is a retry safe to repeat? Is the failure logged with the correlation identifier support will search for?
5. Security. String-built SQL or shell commands, secrets in code or logs, authorization checked on every object and not only on the route, personal data in log lines. The OWASP API risks as review questions are in API Security Testing for Analysts.
6. Fit. Does this belong here, follow the team’s patterns, and reuse what exists? This is where duplicated helpers and outdated idioms surface.

Readability and naming come last, and most of it should already have been handled by the tools.

What should the pipeline automate so humans do not have to?

Everything that has a deterministic answer. If a human reviewer is commenting on formatting, a missing type, or a vulnerable dependency, the pipeline has failed them.

Check	Example tools	Blocks merge?
Formatting	Prettier, Black, gofmt, Spotless	Yes
Lint and deprecations	ESLint, Ruff, Checkstyle, golangci-lint	Yes
Types	TypeScript strict, mypy, pyright	Yes
Secrets	gitleaks, GitHub secret scanning	Yes
Dependencies	Dependabot, Renovate, npm audit, OSV-Scanner	On high severity
Static security analysis	Semgrep, CodeQL, SonarQube	On high severity
Contract diff	oasdiff on the OpenAPI file	On breaking changes
Tests and mutation score	Your suite, Stryker, PIT, mutmut on rule modules	Suite yes, score trend reviewed

An AI reviewer bot can sit on top of this as a first pass: summarising the diff, flagging likely bugs, pointing at missing tests. Treat its comments as bug reports to verify, the same way you would treat a static analysis finding. It does not approve. Approval stays with a person who understood the change and can be asked about it later.

What should the pull request description contain?

Enough that the reviewer can approve without messaging you. This template halves the back and forth on every team I have introduced it to:

## Intent
One or two sentences: what problem this solves and for whom. Link the ticket.

## What changed
- Behaviour changes (what a user, caller, or operator will notice)
- Non-behaviour changes (refactors, renames), or "none"

## How I verified it
- Tests derived from: FR-PAY-12, FR-PAY-13 (rule ids)
- Manual check: submitted a rejected payment in SIT, status endpoint returns RJCT
- Mutation score on status mapping module: 71% -> 88%

## AI assistance
Status mapping and tests drafted with an assistant; mapping table checked
line by line against the scheme status code list. No generated dependencies.

## Risk and rollback
Risk: consumers that treated unknown status as settled now receive PDNG.
Rollback: revert; no migration. Feature flag: payment-status-v2.

The AI assistance section is not a confession. It tells the reviewer where to read for plausibility errors and where to scrutinise the tests for mirror assertions. The risk and rollback section is the one that saves you in production, and it is the same thinking a go or no-go decision needs later.

How should you write review comments?

Label every comment so the author knows what it costs to ignore it. The Conventional Comments format works well:

issue (blocking): Unknown statuses fall through to ACSC. A rejected payment would
report as settled to the customer. Unknown should be an explicit error or PDNG,
never a terminal success status.

question: Is the 30 second timeout from a requirement? The downstream SLA in
the NFR page is 45 seconds at p99.

suggestion (non-blocking): There is already a StatusMapper in payments-core;
reusing it keeps one source of truth for the code list.

nitpick: Test name says "works"; naming it after FR-PAY-12 makes the failure
readable in CI.

Each blocking comment states the consequence, not only the defect. “This is wrong” invites an argument. “A rejected payment would report as settled to the customer” invites a fix. Reviews are also the cheapest coaching channel a team has, which I come back to in Proactivity and Coaching for Developers.

How do you review a diff you do not fully understand?

Ask for locations, not explanations, and verify them. Use the assistant the same way I describe in AI in the Codebase: “which other callers use this function?”, “where else is this status code produced?”, “which configuration values does this read?”. Each answer is a file and a line you can open and confirm, which is safer than a fluent paragraph you cannot check.

Then do the thing that finds most real defects: run it. Check out the branch, call the endpoint with a boundary value and an invalid one, and watch the log. Ten minutes of executing a change beats thirty minutes of reading it, and it is the habit that separates reviewers who catch the silent default from those who approve it. I break down that executable style of review, with API tests you can reuse, in API Testing and QA Mastery for BAs.

How big should a pull request be?

Smaller than the assistant makes easy. Models produce 800-line diffs across fifteen files without effort, and review quality collapses at that size: reviewers skim, approve, and hope. Three rules keep it manageable:

• One intent per pull request. A behaviour change and a refactor are two pull requests, so each can be reviewed for what it is.
• Aim under 400 changed lines, excluding lockfiles and generated code. Beyond that, split or walk the reviewer through it live.
• Generated code is marked and reviewed at its source. If a client is generated from an OpenAPI file, review the OpenAPI change and the generator config, not 3,000 lines of output.

The takeaway

When AI wrote the first draft, code review stops being a proofreading pass and becomes the main control on what enters your system. Keep the author accountable for every line. Review intent, then tests, then boundaries, failure paths, security, and fit. Push everything deterministic into the pipeline. Require a description that says how the change was verified and how to roll it back. Label your comments by consequence, and run the code instead of only reading it.

The silent default in that payment status endpoint would have taken one question to catch: should unknown input have a default at all? Good review is mostly asking that kind of question on purpose.

For where the guardrails on assistant access belong, see AI Agents at Work for Analysts. If you want to set up a review standard for your own team, book a 1:1 Tech BA Coaching Call, or browse everything at The Tech BA Toolkit. The full set of skills this fits into is in The Developer’s Job When AI Writes the Code, and more on reading code lives in the Developer Analyst hub.

Ahmed is a Senior Technical Business Analyst with 10+ years in banking and payments. He builds practical guides and tools for analysts at The Tech BA Toolkit.

Tags: Software Development, Banking, Career Growth, Code Review, Artificial Intelligence