↓Skip to main content

From Code Review to Evidence Review

·8 mins

In October 2025, I wrote The Role of the Reviewer in the Age of LLM-Generated Code. I argued that writing code had become cheap while reviewing it remained expensive. Reviewers were often the first people to understand generated changes properly, so I concluded that their role was more critical than ever.

Generated code without ownership remains dangerous, but capable agents now do much more than propose code changes. I would frame my conclusion differently today: developer judgment is more critical than ever, but traditional code review is no longer always the best place to concentrate it.

The harder question is how engineering organizations can make AI-generated changes measurable, verifiable and safe to deploy. Code review remains one component in a broader verification system and it is increasingly performed by agents rather than humans.

What has not changed #

The main problem from my original article still exists. An engineer asks an agent to implement a task, scans the result, sees passing tests and submits the change. The reviewer becomes the first person to question the design, inspect edge cases and understand how the change fits into the wider system. That workflow outsources judgment to both the model and the reviewer.

Authors still need to understand the changes they submit. Tests still need to validate behavior rather than mirror implementation. Linters, formatters, type checks and static analysis should still remove trivial feedback before a human sees the change. Teams still need explicit standards for what review-ready work looks like.

Agents no longer stop at the repository boundary #

The coding assistants I used when writing the original article were already useful, but they worked mostly within the current codebase. They could generate an implementation, explain unfamiliar logic, run tests and help review a diff.

In my current workflow, agents can follow a problem much further.

They can locate related behavior across repositories, inspect deployed code, check Argo CD state, read Kubernetes pod logs, start a diagnostic pod, query a development database, trigger requests and investigate third-party API behavior. They can move from a failing request to a caching hypothesis, produce a reproduction script, change the implementation and run the relevant checks.

Many difficult bugs do not live cleanly inside one diff. They emerge from interactions between services, deployments, data, concurrency and external systems.

I notice the improvement most in end-to-end and asynchronous work. Race conditions and flaky behavior are still difficult problems, but capable agents can now help run repeated or concurrent reproductions, inspect timing and shared state, compare competing hypotheses and gather evidence from the running system. That is far more useful than producing a plausible static explanation of why the code should work.

The unit of AI-assisted work has shifted from “generate this code” toward “investigate and resolve this engineering problem.” Reviewing the final diff cannot verify work across all those boundaries by itself.

A debugging session that changed my view #

During pre-deployment testing of an internal AI gateway, requests failed in the test environment even though the application path initially looked reasonable. Understanding the failure required application code, deployed configuration, multiple pods, runtime logs and communication between instances.

I started with GPT-5.5 using high reasoning effort. It helped map the relevant path through the system, understand the implementation and narrow the likely source of the failure. The session was useful and I made real progress.

Halfway through the investigation, I used /compact and continued the same conversation with GPT-5.6 Sol. It immediately tested my assumption about communication between multiple gateway pods, found the relevant documentation for the deployed version, generated a small reproduction script and proposed a fix.

I checked the documentation, ran the reproduction and confirmed that communication between the pods did not behave as assumed. The issue remained hidden in development because that environment used one pod. The test environment used multiple replicas to match production. The model identified the relevant system boundary, while I still had to validate the evidence, confirm the root cause and own the decision about the fix.

One debugging task cannot benchmark two models. But in my own work, I noticed clear improvements with each new release, Opus 4.7, then 4.8, GPT-5.5, Claude Fable 5 and GPT-5.6 Sol. During this investigation, pairing with the model felt natural and produced a useful chain of evidence:

  • a failure observed during realistic pre-deployment testing
  • runtime evidence from the deployed system
  • a concrete hypothesis about cross-pod behavior
  • external source material supporting that hypothesis
  • a minimal reproduction script
  • confirmation in the relevant environment
  • a proposed change tied to the confirmed cause

and by now a newly released verion of the dependency that contains the fix.

A reviewer could still inspect the eventual code change line by line, but the diff alone could not establish most of these facts. Reconstructing them becomes even harder when the reviewer did not participate in the investigation.

The bottleneck moved from review to verification #

The ‘old’ bottleneck was straightforward:

Who has time to read, understand and verify all this generated code?

The more important question now is:

What evidence shows that this change behaves correctly under realistic conditions?

Generated code can be coherent and wrong. It can satisfy unit tests while failing under concurrent load, work in one pod while failing across replicas or misunderstand a third-party service’s actual rate limits.

Reading the diff and running tests can help identify these risks, but they cannot cover them all.

Blind trust in automated checks would reproduce the same problem. Tests can encode the wrong assumptions, benchmarks can measure the wrong workload and AI reviewers can miss the same context as AI implementers (or as humans). A green pipeline can create false confidence as easily as a plausible diff. Verification therefore needs to become an engineering discipline of its own.

That includes deterministic checks such as types, tests and static analysis, but it also includes representative data, end-to-end workflow testing, performance measurements, migration validation, tests for timeouts, dependency outages and recovery, concurrency testing and deployment checks. Which evidence matters depends on the change.

The engineer identifies which assumptions carry meaningful risk and produces enough evidence to justify trust. Running every possible check would add cost without improving every decision.

From code review to evidence review #

For non-trivial work, sufficient evidence has to accompany the diff.

Depending on the change, that might include:

  • the problem, intended behavior and non-goals (usually a detailed ticket or user story)
  • important constraints and assumptions
  • a recording of the validation process when the behavior is easier to show than describe
  • a matrix, flowchart or other diagram when it clarifies behavior before and after the change
  • alternatives considered and the selection rationale
  • tests added, the behaviors they protect and why those tests are sufficient
  • representative workflows or data used during validation
  • relevant performance, concurrency or failure evidence
  • deployment considerations, rollback options and remaining risks

The greater the risk, the stronger the evidence should be. A small, reversible fix may need two sentences and one regression test. An irreversible data migration or concurrency-sensitive feature should demand much stronger evidence.

The reviewer can still inspect any part of the diff. But the first question changes from:

Does this implementation look correct?

to:

Is the intent clear and is the evidence sufficient for the risk of this change?

That question directs human attention toward missing assumptions, weak experiments, architectural consequences and product suitability. It also makes review less dependent on one person reconstructing the entire development process from the final code.

What human review becomes #

Human review remains important for security boundaries, irreversible migrations and decisions with significant architectural or product consequences. Routine review work should move elsewhere.

Formatting, imports, naming conventions, obvious defects and predictable repository rules belong in automated checks and focused AI review. Missing tests should be discovered while designing validation, not after implementation reaches a reviewer. Runtime assumptions should be tested in representative environments rather than debated only in comments.

Human review can focus on questions that require judgment:

  • Does this solve the right problem?
  • Which assumptions carry the most risk, and does the evidence cover what could go wrong?
  • Is the change reversible and aligned with the system’s direction?
  • What should the team learn and preserve from this work?

For larger changes, a short engineering walkthrough may transfer more knowledge than scattered inline comments. The author can explain the problem, alternatives, evidence, remaining risk and final rationale. An agent can prepare much of that summary from the development history, but the engineer must read, validate and own it before asking others to review it.

Human review should build shared understanding, apply engineering judgment and support organizational learning instead of serving as the only place where defects can be found. A useful standard for AI-generated changes is reviewable, measurable and reversible, regardless of diff size.

Trust requires evidence #

My interpretation is that engineering organizations cannot scale trust by asking humans to inspect every generated line with equal intensity. As agents perform more implementation and investigation work, teams need clearer intent, stronger automated verification, realistic workflow evidence and engineers who remain accountable for the result.

My prediction is that automated verification will become one of the most valuable forms of engineering infrastructure. Every realistic end-to-end scenario, regression check, concurrency harness, deployment validation and reproduction of a known failure makes future agent-generated work safer. Better models increase the leverage of those assets, but do not remove the need for them.

Engineers will increasingly build both features and the verification systems that let agents iterate toward correct outcomes. I explore this in Building the Systems That Produce Software.

Authors remain responsible for explaining the intent, evidence and rationale well enough for reviewers to build shared understanding. Reviewers apply judgment and challenge what is missing without serving as the entire safety net. As code production gets cheaper, trust must come from evidence and clear accountability rather than more time spent on human inspection.