What is metamorphic testing?

What is metamorphic testing?

Metamorphic testing is a technique that runs a program on related inputs and checks that their outputs relate to each other by a known rule. The rule is called a metamorphic relation, e.g. narrowing a search can only remove results. The test compares the outputs with each other, so nobody needs the exact correct output for either input.

Most tests compare an output with an expected value from a test oracle. Metamorphic testing is for code whose correct output is hard to know in advance, e.g. search results from a large catalog. Its tests are often unit tests that call one function twice. When nobody can write down the expected values, tests for such code can reach high code coverage and still check little.

The International Software Testing Qualifications Board (ISTQB) defines metamorphic testing as a test technique whose test conditions are metamorphic relations. Its Advanced Test Analyst and AI Testing syllabi cover the technique. Chen, Cheung, and Yiu introduced it in a 1998 technical report as a technique that "can be used in the absence of test oracles."

How does metamorphic testing work?

A metamorphic test is written and run in five steps:

1. A person writes a metamorphic relation from the requirement.
2. The test runs the program on a source input and records the first output.
3. The test changes the source input as the relation describes.
4. The test runs the program on the changed input and records the second output.
5. The test checks both outputs against the relation and fails if they break it.

Any input that meets the relation's conditions can be the source, so one relation checks many pairs of runs. Relations come from the requirements and the problem domain, not from the code. A developer asks which input changes have a known effect on the output. Relations often follow these patterns:

• Symmetry. Swapping parts of the input keeps the output, e.g. on roads that run both ways, the shortest trip from A to B is as long as the one back.
• Reordering. Reordering input that has no meaningful order keeps the output, e.g. adding cart items in another order keeps the subtotal.
• Narrowing. Adding a condition to a query can only remove results, e.g. a search in one category returns a subset of the results for the whole store.
• Scaling. Scaling the input changes the output in a known way, e.g. doubling each item's price doubles the cart's subtotal.
• Irrelevant change. A change that should not affect the result leaves it unchanged, e.g. an export command prints the same rows whether it writes to a terminal (TTY) or a pipe.

What is an example of a metamorphic relation?

Here is an illustrative example. Acme Co. sells furniture online, and its web store has a product search. A developer at Acme asks a coding agent to "Let customers filter search results by price." The agent adds a max_price option and a test that checks that each filtered result costs $300.00 or less. That test passes.

Nobody at Acme can write down the exact results of each query, because they depend on the catalog and ranking rules. The developer writes a relation from the request instead. Adding a price filter can only remove results, never add them. The test runs the search twice and compares the outputs:

from acme.search import search

def test_price_filter_only_removes_results():
    everything = set(search("oak desk"))
    filtered = set(search("oak desk", max_price=300))
    assert filtered <= everything

The test fails with this output:

>       assert filtered <= everything
E       AssertionError: assert {'D-112', 'D-207'} <= {'D-104', 'D-112', 'D-118'}
E
E         Extra items in the left set:
E         'D-207'

The filtered search returned D-207, a discontinued oak desk at $240.00. The unfiltered search hides discontinued products, but the agent's filter read the whole catalog. The agent's own test passed because the desk does cost less than $300.00. The relation caught the bug without a list of expected results.

The agent changes the filter so it narrows the normal search results. The developer keeps the test as a regression test and runs it on more queries. This example is simplified. A real store would need more relations, e.g. one for sorting by price.

How does metamorphic testing help with the oracle problem?

The test oracle problem is the difficulty of deciding whether a test passed or failed. It is hardest when nobody knows the correct output. Metamorphic testing handles part of the problem by replacing the missing expected value with a relation. The relation states only how the outputs of related runs must relate. That makes it a partial oracle, which checks some properties of a result but not the whole result.

The original report's hard case is checking that a path through a large graph is the shortest. When the graph's roads run both ways, a test can still check the symmetry relation on many pairs of places. Machine learning is another common use, because the correct prediction for an unseen input is often unknown or costly to find. A relation can require that an image model gives a slightly brighter photo the same label.

Some teams ask a large language model (LLM) to grade output against a rubric instead, a setup called LLM-as-a-judge. The judge's score can vary between runs, while a relation gives the same two outputs the same verdict each time.

What changes when a coding agent writes the code?

A coding agent that writes a function and its tests can take the expected values from the function, e.g. by pasting the code's output into the test. A wrong function then produces a wrong expected value, and the test passes. A metamorphic relation has no expected value to copy. It is a rule that follows from the request, e.g. "a price filter can only remove results." A reviewer can check that rule against the request without working out any search result.

Agents can also write relations that are valid but weak. Running the same search twice and expecting the same results catches only results that change at random. The price filter relation in the example is stronger, because its second run goes through the filter code that the agent added.

A practical adjustment is to write at least one relation per behavior the change adds. For each relation, a reviewer asks which plausible bug would break it. If no bug would, the relation checks too little.

What are the limits of metamorphic testing?

A passing metamorphic test shows that its relations held for the inputs that ran. It does not show that the code is correct. The technique has five limits:

• A relation is necessary, not sufficient. In a 2018 survey, Chen and colleagues describe metamorphic relations as necessary properties of the function under test. Code can satisfy a relation and still be wrong, e.g. a filter that returns no results passes the subset check.
• Finding relations takes domain knowledge. A developer has to reason about the problem, and a weak relation passes on many bugs.
• Each check costs extra runs. A relation needs at least two runs, so a metamorphic test takes longer than a test with one expected value.
• Varying output needs a looser relation. When outputs vary between runs, the relation must allow for it, e.g. tied results in a ranking that swap places.
• Relations cover only what someone thought to relate. Exploratory testing by a person can find behavior that no relation describes, e.g. a filter that ignores sale prices.

How is metamorphic testing different from property-based testing?

Property-based testing usually checks a rule about one run, often an invariant, e.g. an order total is never negative. A metamorphic relation links two or more runs, e.g. the filtered results are a subset of the unfiltered results.

The two overlap when a property-based framework, e.g. Hypothesis, generates the source inputs for a property that checks a metamorphic relation. Differential testing, a third neighbor, runs one input through two or more implementations instead of related inputs through one.

FAQs

What is a metamorphic relation?

A metamorphic relation is a rule that states how a program's output should change, or stay the same, when its input changes in a known way. Adding a price filter to a search can only remove results, so a filtered search that returns an extra product breaks that relation.

Is metamorphic testing used for machine learning?

Metamorphic testing is used for machine learning, because the correct prediction for an unseen input is often unknown or costly to find. A relation can require that a slightly brighter photo keeps the model's label. A passing relation shows that the label stayed consistent, not that it was correct.

How do you find metamorphic relations?

Metamorphic relations come from the requirements and the problem domain, not from the code. A developer lists changes to an input whose effect on the output is known, e.g. swapping the start and end of a trip. The common patterns are symmetry, reordering, narrowing, scaling, and irrelevant changes.

Who invented metamorphic testing?

Chen, Cheung, and Yiu introduced metamorphic testing in a 1998 technical report. They proposed it as a way to derive test cases from ones that passed, and noted that it works without a test oracle. Chen and colleagues reviewed the research that followed in a survey.