Verifying AI-written code
How to check that software written by coding agents works, from "done" claims and gamed tests to evidence from running the code.
Start here
How to test AI-generated code
Testing AI-generated code means checking what the software does, not who wrote it, by reading the change, running its tests, and running the app itself.
Concepts
What is runtime verification?
Runtime verification is a method that checks a running program against stated properties by observing its behavior, instead of analyzing its source alone.
What are AI hallucinations in code?
AI hallucinations in code are imports, calls, flags, or options that name a package or API that does not exist, generated in the same form as real code.
What counts as evidence that code works?
Evidence that code works is a record of the software doing the right thing under stated conditions, with the command, its output, and the expected result.
What is a definition of done for coding agents?
A definition of done for coding agents lists the evidence a change must produce, e.g. a passing run of the app, before the work counts as finished.
What is a holdout test suite for coding agents?
A holdout test suite is a set of hidden tests kept out of a coding agent's reach, so they check its final work without the agent reading or editing them.
What is AI slop in code?
AI slop in code is AI-generated output that looks finished but is bloated, duplicated, or poorly tested, and it moves the work of checking onto reviewers.
What is independent verification and validation (IV&V)?
Independent verification and validation (IV&V) is review, analysis, and testing by a party outside the development team, to limit shared assumptions.
What is LLM-as-a-judge?
LLM-as-a-judge is the use of a language model to grade outputs against a rubric, which scales review but adds biases and does not run the software.
What is reward hacking in coding agents?
Reward hacking is when a coding agent satisfies the check it is graded on, e.g. the test suite, without doing the task the user intended.
What is SWE-bench, and what does a score prove?
SWE-bench is a benchmark that gives coding agents real GitHub issues and scores each patch with the project's own tests, which check only part of the fix.
What is verification debt?
Verification debt is the growing gap between how much AI-written code a team produces and how much of it anyone has checked by running or testing it.
How-to guides
How to stop coding agents from editing tests
Stopping a coding agent from editing tests means blocking its writes to test files, reviewing test changes, and running the tests outside its session.
How to verify code you cannot read
Verifying AI code without reading it means checking what the running app does against results written in advance, and keeping a record of each check.
Comparisons
What is the difference between reading code and running code?
Reading code, as in AI code review, finds problems visible in the source, while running it, as in testing, finds failures that appear only at runtime.
What is the difference between verification and validation?
Verification checks software against its specification, while validation checks it against users' needs, and neither is defined by whether the code runs.
Common questions
Why do coding agents say "done" when the code doesn't work?
A coding agent's "done" is a completion claim written by the same process that wrote the code, so it can arrive with no passing run behind it.
Why do coding agents delete or weaken tests?
Test tampering is a coding agent behavior that edits, skips, or deletes a failing test so the suite passes, instead of fixing the code it checks.
Can AI check its own code?
AI self-verification is a model reviewing or testing its own output, a useful first pass whose checker shares the author's context and blind spots.
Is AI-generated code secure?
AI-generated code carries the same security risks as code people write, plus a few of its own, so it needs review, scanning, and testing before release.
Why do AI-written tests always pass?
AI-written tests often pass because they assert what the code already does, mock the parts that fail, or check almost nothing, so a green run shows little.
Why doesn't "no bugs found" mean "no bugs"?
"No bugs found" means only that nothing failed in the checks that ran, so a useful report also names what it skipped, did not reach, or could not decide.
Failures and bugs
What is a false pass (false green test)?
A false pass is a test or check that reports success while the software is broken, because the check missed the defect or the failure was lost.
Common bugs in AI-generated code
Common bugs in AI-generated code are recurring failures that a quick demo does not reveal, from open data access to features that never run.
What are out-of-scope edits by coding agents?
An out-of-scope edit is a change a coding agent makes to code its task did not ask for, e.g. an unrequested refactor, and it can break other features.
Terms in this topic
- AI code verification
- AI code verification is the practice that checks what AI-generated software does, by reading it, running its tests, and running the software, rather than detecting who wrote it.
- AI slop
- AI slop is AI-generated output that looks finished but is bloated, duplicated, or poorly tested because nobody checked it closely. In code it includes verbose logic and tests that assert almost nothing.
- Assertion-free test
- An assertion-free test is a test that runs code but checks no result, so it passes whenever the code does not crash, whether the output is right or wrong.
- Benchmark contamination
- Benchmark contamination is a flaw in which a benchmark's tasks or their solutions reach a model through its training data or its tools, so a score can reflect recall or lookup of a known fix rather than skill.
- Cross-model verification
- Cross-model verification is a checking setup that asks a different AI model to review work that another model produced, in the hope that their blind spots differ.
- Definition of done
- A definition of done is a shared checklist that states what evidence a piece of work must have before a team counts it as finished.
- Eval
- An eval, short for evaluation, is a repeatable test of an AI system that runs a fixed set of tasks and scores the results, so versions of the system can be compared.
- False completion claim
- A false completion claim is a coding agent's report that a task is "done," or that tests pass, when the work has not been built, run, or checked.
- False negative
- A false negative is a test result that misses a defect that does exist, so the check passes even though the software is broken.
- False pass
- A false pass is a test or check result that reports success while the software is broken, because the check missed the defect or lost the failure.
- Ghost feature
- A ghost feature is a feature that exists in the code and may pass its tests but never runs for users, e.g. a handler that nothing calls.
- Holdout test suite
- A holdout test suite is a set of tests kept out of a coding agent's reach, so they can check the agent's finished work without the agent reading or editing them first.
- Inconclusive result
- An inconclusive result is a check outcome that could not finish or could not decide, so it is neither a pass nor a fail and should be reported as such.
- Independent verification and validation (IV&V)
- Independent verification and validation (IV&V) is review, analysis, and testing done by a party separate from the developers, so that the check is less likely to share the developers' assumptions.
- LLM-as-a-judge
- LLM-as-a-judge is an evaluation setup that uses a language model to grade output against written criteria, instead of or alongside deterministic checks.
- Out-of-scope edit
- An out-of-scope edit is a change that a coding agent makes outside what its task asked for, e.g. an unrequested refactor.
- Package hallucination
- Package hallucination is a code generation failure that imports or installs a package that does not exist. An attacker can register the invented name and publish harmful code under it.
- Reward hacking
- Reward hacking is a behavior in which a coding agent satisfies the check it is graded on, e.g. the tests, without doing the task the user intended.
- Row-level security
- Row-level security is a database feature that limits which rows each user can read or change, so one user's data stays hidden from another.
- Runtime verification
- Runtime verification is a method that checks a running program against stated properties by observing its behavior, usually through a monitor that reads its execution. Developers also use the term for running the software to check a change.
- Slopsquatting
- Slopsquatting is an attack that registers package names that language models tend to invent, so code that installs a hallucinated package installs the attacker's code instead.
- Specification gaming
- Specification gaming is a behavior in which an AI system satisfies the literal objective it was given without doing the task the objective was meant to capture.
- SWE-bench
- SWE-bench is a benchmark that gives coding agents real issues from open-source Python projects and counts an issue as resolved when the project's tests pass after the agent's patch.
- Test evidence
- Test evidence is a record of a check that ran, including what ran, on which version of the code, and what was observed, so another person can review or repeat it.
- Test tampering
- Test tampering is a coding agent behavior that edits, skips, or deletes a failing test so the suite passes, instead of fixing the code the test was checking.
- Verification debt
- Verification debt is the gap that grows between how much AI-written code a team produces and how much of it anyone has checked by running or testing it.