What is assertion coverage?

What is assertion coverage?

Assertion coverage is a test measure that counts the code whose results the tests check with an assertion, rather than all the code that runs. A statement that runs during a test but never affects a checked value adds to line coverage and not to assertion coverage. Researchers call one precise version checked coverage.

Assertion coverage has no standard definition, and authors use the name for several related measures. In a 2011 paper, David Schuler and Andreas Zeller defined checked coverage as the statements that ran and also influenced a value that a check tested. In hardware design verification, the same phrase names a different measure, which counts whether a chip design's assertions were exercised in simulation.

An assertion is a line of test code that checks a condition about a result and fails the test when the condition is false. Line coverage is the share of lines that ran during the tests, and code coverage tools report it without reading any assertion. The gap between line coverage and assertion coverage is code that a unit test runs but that could return a wrong result without any test failing.

How is assertion coverage measured?

A checked coverage tool measures one traced run of the test suite in four steps:

1. The tool finds each check in the test code, e.g. each call to an assert method.
2. The tool runs the tests once and records which variables each statement reads and writes, and each branch it takes.
3. From each check, the tool follows those records backward to the statements that produced the checked value or decided whether it was computed. This set is called a dynamic backward slice.
4. The tool divides the number of statements in the slices by the number of statements that could run.

Schuler and Zeller built their tool for Java on a research slicer. Common coverage tools report the lines and branches that ran, not checked coverage. Teams without a slicer use proxies that measure part of it:

• No assertion at all. Jest's expect.hasAssertions() fails a test that runs no assertion, and the jest/expect-expect rule in eslint-plugin-jest flags a test with no expect call.
• Mutation score over covered code. Mutation testing makes small changes to the code and checks whether a test fails. PIT calls the share of changes in covered code that a test caught its test strength.
• Methods whose bodies no test checks. Descartes, an open-source engine for PIT, empties the body of a method or replaces it with a single return of a fixed value, e.g. null. When a method's tests still pass after each such change, the method runs but no test checks what it does.

What is an example of low assertion coverage?

Here is an illustrative example. Acme Co. sells furniture online. A developer asks a coding agent to "Let customers edit their delivery address during checkout." The agent writes this function and a pytest test:

def change_address(order, new_address):
    address = new_address.strip()
    if not address:
        raise ValueError("delivery address is empty")
    items = [(sku, qty) for sku, qty in order.items if qty > 1]
    return Order(id=order.id, address=address, items=items)


def test_change_address():
    order = Order(id="A-1042", items=[("sofa", 1), ("lamp", 1)])
    updated = change_address(order, "12 Elm Street")
    assert updated.address == "12 Elm Street"

The test passes. A coverage report shows that 4 of the 5 statements in the function body ran. Only the raise line, for a blank address, did not run.

A reviewer traces the assertion back through the code, the way a checked coverage tool does. The checked address depends on the strip line, on the if that let the function reach its return, and on the return itself. No assertion depends on the items line. Line coverage counts 4 of 5 statements, and assertion coverage counts 3.

The unchecked line holds the bug. It is meant to drop cart lines with a quantity of 0, but it keeps only lines with a quantity above 1. The address saved, but the cart emptied. One added assertion puts the items line on a checked path, and the test fails:

>       assert updated.items == order.items
E       AssertionError: assert [] == [('sofa', 1), ('lamp', 1)]

Line coverage did not change. Assertion coverage rose to 4 of 5, the same as line coverage. This example is simplified. A real project would need a tool to trace thousands of statements across many files.

What changes when a coding agent writes the code?

AI-written tests can check only the result the prompt names, e.g. the delivery address, and leave the rest of the change unchecked. Such a test has an assertion, so the proxies that flag a missing assertion pass it. Line coverage counts each changed line that ran. Assertion coverage counts only the lines behind that one check.

A 2026 study of 86,156 changes to test files by five coding agents found that 80.2% had weak or no explicit oracle signals. Many of those tests ran code while checking little or nothing about its output. The authors classified the checks by matching patterns in the added lines of each patch, without running the tests or tracing which statements each check reached.

A filter for AI test case generation that keeps tests which pass and add line coverage never reads what they check. A practical adjustment is to trace the diff by hand when reviewing agent pull requests, as in the Acme example. For each changed line, name the assertion that depends on its result, and ask for one with an expected value from the request where none does. Checking agent-written tests with deliberate bugs then confirms that the added checks fail.

What are the limits of assertion coverage?

Assertion coverage shows which code a check depends on, not whether the check's test oracle is right. A suite with high assertion coverage can still give a false pass, a green run on broken software. The measure has four limits:

• A weak check still counts. In Schuler and Zeller's paper, one check that a parser's result was not null put most of the parser on a checked path.
• A wrong expected value still counts. A tautological test builds its expected value from the code under test, so it raises assertion coverage and passes on wrong code. Mutation testing exposes it, because a mutant, a copy of the code with one small bug, changes both sides of the check and the test still passes.
• Missing code has no statement to check. When the code never handles a case, neither line coverage nor assertion coverage has a line to report.
• Tracing is slow. A checked coverage tool records each read, write, and branch. In Schuler and Zeller's study, tracing and computing the slices took far longer than running the same tests.

How is assertion coverage different from line coverage?

Line coverage counts the statements that ran during the tests. Assertion coverage counts the part of that set whose results an assertion depends on, so it cannot be higher than line coverage over the same statements. In Schuler and Zeller's study of open-source Java projects, checked coverage was well below statement coverage in each project. The two measures differ on these points:

Mutation score is a third measure, which counts injected bugs that made a test fail, including crashes that no assertion checks.

FAQs

What is checked coverage?

Checked coverage is the research form of assertion coverage that Schuler and Zeller defined. A tool traces one run of the test suite and follows each assertion back to the statements that influenced the checked value. The score is the share of those statements among all statements that could run.

Which tools measure assertion coverage?

Few common tools measure assertion coverage directly, and checked coverage came from a research tool for Java. Teams use proxies instead, e.g. a lint rule that flags a test with no assertion. Mutation testing tools add test strength and a list of methods whose bodies can be emptied without a test failing.

Does assertion coverage replace mutation testing?

Assertion coverage does not replace mutation testing, because the two measures answer different questions. Assertion coverage shows which code feeds an assertion, and mutation testing shows whether a test fails when that code changes. A test that builds its expected value from the code raises assertion coverage and still lets mutants survive.

Can a coding agent game assertion coverage?

A coding agent can game assertion coverage by adding checks that almost any result passes, e.g. a check that a result is not null. An expected value built from the code under test also counts. The number rises while the tests catch no more bugs, so a reviewer still reads each expected value.