Test quality and test oracles

Terms in this topic

Adversarial testing

Adversarial testing is a testing approach that deliberately tries to break software with unexpected, hostile, or careless inputs and actions, instead of confirming the expected path.

AI test case generation

AI test case generation is the use of a language model to write test cases from code or a specification. Some tools then drop tests that fail to build, fail, flake, or add no coverage.

Assertion

An assertion is a statement in a test that checks a condition and fails the test when the condition is false.

Assertion coverage

Assertion coverage is a test measure of how much of a program's behavior the tests check with assertions, as opposed to how many lines run while the tests execute.

Branch coverage

Branch coverage is a code coverage measure that counts which outcomes of each decision point, e.g. both sides of an if statement, the tests have run.

Catching test

A catching test is a test that is generated for one code change and designed to fail if that change introduced a bug, then usually thrown away after review.

Chaos engineering

Chaos engineering is the practice of running controlled experiments that inject faults into a system, e.g. a stopped server, to learn how the system copes before a real outage.

Characterization test

A characterization test is a test that records what existing code does now, right or wrong, so later changes can be checked against that recorded behavior.

Code coverage

Code coverage is a test measure that reports the percentage of a program's code that runs during its tests, which shows what was executed but not what was checked.

Coverage-guided fuzzing

Coverage-guided fuzzing is a fuzzing method that keeps the inputs which reach new code paths and mutates them further, so the fuzzer explores a program efficiently.

Differential testing

Differential testing is a technique that gives the same inputs to two or more comparable implementations, or two versions of one program, and treats any difference in output as a possible bug.

Equivalent mutant

An equivalent mutant is a code change made during mutation testing that behaves exactly the same as the original program, so no test can ever detect or kill it.

Fail-before, pass-after check

A fail-before, pass-after check is a test of a test that must fail on the code before a change and pass on the changed code.

Fault seeding

Fault seeding is a technique that deliberately inserts known bugs into code to see how many of them a test suite or review process detects.

Fuzzing

Fuzzing is a testing technique that feeds a program large numbers of random or malformed inputs to find crashes, hangs, and bugs in input handling.

Golden file

A golden file is a saved, approved copy of a program's output that a test compares new output against, failing when anything in the output changes.

Implicit oracle

An implicit oracle is a general failure signal that marks a result as wrong without any expected value, e.g. a crash.

Incremental mutation testing

Incremental mutation testing is mutation testing that tests only the mutants a change could affect, e.g. those in the lines of a pull request, instead of every mutant in the codebase.

Metamorphic testing

Metamorphic testing is a technique that checks how a program's output should change when its input changes in a known way, so code can be tested without an exact expected output.

Mutant

A mutant is a copy of a program with one small deliberate change, e.g. a flipped comparison, that mutation testing uses to check whether the tests fail.

Mutation score

Mutation score is a test-strength measure that reports the share of valid mutants a test suite detects, which shows how well the tests check behavior.

Mutation testing

Mutation testing is a technique that checks a test suite by injecting small deliberate bugs, called mutants, and counting how many of them the tests catch.

Partial oracle

A partial oracle is a test oracle that checks some properties of a result when the full expected result is unknown, e.g. that an order total is never negative.

Property-based testing

Property-based testing is a technique that states rules that should hold for every input, then checks them against many generated inputs and shrinks any failing input.

Shrinking

Shrinking is a step in property-based testing that reduces a failing generated input to the smallest input it can find that still fails, so the bug is easier to understand.

Tautological test

A tautological test is a test that cannot fail because its expected value comes from the same code, mock, or constant as the result it checks. It passes whether the code is right or wrong.

Test oracle

A test oracle is the source of truth that a test compares a result against, e.g. a specification that states the correct output for a given input.

Test oracle problem

The test oracle problem is the difficulty of deciding whether a test passed or failed for a given input and state, which is hardest when nobody knows the correct answer.

28 terms from the Learning Center glossary.