Why doesn't "no bugs found" mean "no bugs"?
"No bugs found" does not mean "no bugs," because a test run can report a failure only in what its checks ran and compared. Skipped tests, checks that stopped early, and behavior that no check reached add nothing to the summary. A report that names what it did not check shows how far its clean result reaches.
In his 1972 Turing lecture, Edsger Dijkstra wrote that testing can show the presence of bugs but "is hopelessly inadequate for showing their absence." The International Software Testing Qualifications Board (ISTQB) states the same idea as the first testing principle of its Foundation Level syllabus. A quiet run is an absence of evidence about the parts it did not reach, not evidence that those parts work.
The gap is harder to see when a coding agent writes the code, runs the checks, and writes the report, because a reader often gets only its summary. Checking what a run skipped is part of how teams test AI-generated code.
What causes a clean result that hides gaps?
A clean result hides gaps in four ways. Each one leaves the summary line reading "no bugs found."
Why does a clean run speak only for what it ran?
Software testing runs a sample of what the software can do, because most programs accept too many inputs to try them all. The ISTQB defines coverage as the degree to which a test suite exercises specified coverage items, as a percentage. Full coverage of those items says nothing about behavior outside them.
SWE-bench, a benchmark built from real GitHub issues, checks each patch by running only the test files that the original pull request changed. A 2025 study found that, on average, 7.8% of the AI patches that SWE-bench's checks counted as correct failed the developers' own tests.
How do checks that never ran or never finished look clean?
A skipped test is marked not to run, and many runners list it in the summary and still exit with code 0. A suite in a separate script, e.g. browser tests, never starts unless someone runs that script. In GitHub Actions, a job skipped by its if condition reports its status as "Success," so it does not stop a pull request from merging, even as a required check.
A check that crashes or times out before its first comparison has compared nothing, and a summary that counts only failures shows none for it.
Why can a check run and still miss the bug?
A check compares a result only with its test oracle, the source of the expected result. A test that checks the delivery address and not the cart passes on code that empties the cart. The ISTQB calls that outcome a false negative result, a test result that misses a defect that is present. Developers also call it a false pass. Code coverage still counts the cart code as covered, because it records which lines ran, not which results were checked.
How does a summary lose the scope?
A summary compresses a run into one line, e.g. "no bugs found." The counts, the skips, and any failure that retries turned into a pass drop out of that line. A coding agent often writes the line at the end of its session, and the line can report a clean result for checks that never ran. Such a report is a false completion claim.
What does a report with hidden gaps look like?
Here is an illustrative example. Acme Co. sells furniture online. A developer asks a coding agent to "Let customers edit their delivery address during checkout." The agent changes the checkout code and ends its session with this report:
Done. All checks passed. No bugs found.
npm test 14 passed, 1 skipped
lint 0 problems
The report is accurate about what ran. The developer reads the run output and the project's scripts and finds three gaps:
- The developer reads the 14 passing unit tests. They check the address form, e.g. that it saves 12 Elm Street, and none of them checks the cart.
- The developer finds the skipped test,
payment page shows the cart total, which runs only whenPAYMENT_SANDBOX_KEYis set, and the agent's environment did not set it. - The developer finds the browser tests in a separate script,
npm run test:browser, which nothing ran.
The developer then adds a coverage note to the report:
Passed: address form saves the address (14 unit tests), lint (0 problems)
Skipped: payment page cart total (PAYMENT_SANDBOX_KEY not set)
Unchecked: checkout in a browser (npm run test:browser never ran)
Inconclusive: none
When the developer sets the key and runs both, the skipped test fails and so does the browser suite. The address saved, but the cart emptied. This example is simplified. A real report would also name the commit the checks ran against.
How can teams report what was not checked?
A coverage note is a short part of a test report that says what the checks reached and what they did not. It has four parts, one for each state of a result:
- What passed. The note names each command or suite that ran and passed, and the version of the code it ran against.
- What was skipped. The note lists each skipped test with the reason it did not run.
- What stayed unchecked. The note names each suite that never started and each part of the request that no test covers, e.g. the cart after an address change.
- What was inconclusive. The note lists each check that crashed or timed out before it compared a result.
An empty part says "none," so a reader can tell it from a missing part. Each line should point to its test evidence, e.g. the log of the run, so a reviewer can check it. A short session of exploratory testing can reach workflows that no scripted test covers, and its notes join the report.
When a coding agent writes the report, a team can ask for the commands it ran and their raw output, and can treat an unexplained skip as a failure. A reviewer then weighs the note against the request and the risk of the change. A coverage note lists only the gaps that someone can name, so behavior that nobody thought to check stays off it.
How is an inconclusive result different from a pass?
An inconclusive result is a check outcome that could not finish or could not decide, so it counts as neither a pass nor a fail. A pass compared a result with the expected one and found a match, while an inconclusive check reached no verdict. Reporting it as a pass hides a gap, and reporting it as a failure sends someone to debug a defect that may not exist.
NUnit records the state directly, with Assert.Inconclusive for a test that "could not be completed with the data available." Many monitors in runtime verification return inconclusive while the run so far could still end either way. The four states differ in what they show:
| Result | What happened | What it shows |
|---|---|---|
| Passed | The check ran and the result matched | The checked results were right for the inputs that ran |
| Skipped | The check was marked not to run | Nothing about the behavior |
| Unchecked | No check ran against the behavior | Nothing about the behavior |
| Inconclusive | The check started but could not finish or decide | Nothing yet, so it needs a rerun |
What should a verification report leave you with?
A verification report should leave you knowing how far its clean result reaches and which gap to close next. When the largest gap is a workflow that nobody ran, e.g. checkout in a browser, close it by running the software, not only its tests.
RunStory runs your software in a separate environment, tries relevant workflows, and checks the results. You keep working. It is in private alpha for CLIs and web apps, and your team keeps the final release decision.
FAQs
What should a tester report after finding nothing?
A tester who finds nothing should report "no failures in the checks that ran," not "no bugs." The report names each command, the version of the code it ran against, and each skipped, unchecked, or inconclusive part.
When is a clean result enough to release?
A clean result is enough to release when its coverage note shows that the checks reached the behavior the change can affect and checked its results. No skipped or inconclusive check should be left unexplained, and the team weighs any remaining gap against the risk of the change.
What does absence of evidence mean in testing?
Absence of evidence in testing means that no check reported a failure, which differs from evidence that the software works. Skipped, unchecked, and inconclusive parts add no failure to the count without checking the behavior.
How do you describe coverage gaps in a test report?
Coverage gaps in a test report are described one line each, with the reason for each gap. You name the condition that skipped a test, e.g. a missing environment variable, and the command for a suite that nobody ran. You write "none" in an empty part, so a reader can tell it from a part that was left out.