What is acceptance testing (UAT)?

What is acceptance testing (UAT)?

Acceptance testing is the test level that checks software against the needs of its users and owners, so they can accept or reject it. It usually comes last, before release, and its checks come from agreed acceptance criteria. It is often called user acceptance testing (UAT), though UAT is strictly the form that intended users run.

The International Software Testing Qualifications Board (ISTQB) defines acceptance testing as a test level that focuses on determining whether to accept the system. Its entry for user acceptance testing narrows the question to whether the intended users accept it. Other forms ask the same question for other groups, e.g. the team that will operate the system.

Among the levels of software testing, acceptance testing follows system testing. The ISTQB Foundation Level syllabus says that it focuses on validation. The ISTQB defines validation as confirmation that a work product matches a stakeholder's needs. Acceptance testing is therefore the level that asks whether the team built the right thing, one half of verification and validation.

How does acceptance testing work?

Acceptance testing turns each agreed criterion into a check on the finished software. For one change, it usually takes six steps:

1. The people who asked for the change agree on acceptance criteria with the team before the build starts.
2. A tester or a developer turns each criterion into a manual or automated test case with inputs and an expected result.
3. The team builds the change and runs its own tests at the lower levels.
4. The acceptance tests run on the complete software in an environment close to production, e.g. a staging copy.
5. The team records the result of each test next to its criterion.
6. The owners read the results, try the software if they choose, and accept the change or send it back with the defects they found.

When the team writes the acceptance tests before the code, the practice is called acceptance test-driven development (ATDD). The ISTQB describes ATDD as a collaborative approach that writes acceptance tests first, in the language of the stakeholders. The tests fail until the code meets the criteria. On agile teams, this loop runs for each user story instead of once per release.

Automated acceptance tests usually drive the software through its user interface or its public API. Many of them are end-to-end tests, and some follow one whole task as user journey tests. A team can run them on each change and make them a quality gate that a release must pass.

What is an example of an acceptance test?

Here is an illustrative example. Acme Co. sells furniture online. Its product owner asks the team to "Let customers edit their delivery address during checkout." Before any code exists, the owner and a developer write the acceptance criteria as a Given-When-Then scenario:

Scenario: Customer edits the delivery address at checkout
  Given a customer has 2 items in the cart
  And the customer is on the checkout review step
  When the customer changes the delivery address to "12 Elm Street"
  Then the review step shows "12 Elm Street"
  And the cart still holds 2 items

The change then goes through acceptance in six steps:

1. The developer automates the scenario as a browser test, with setup steps from the Given lines and checks from the Then lines. It runs against staging.shop.example.com, a staging copy of the store.
2. A coding agent builds the change and reports that the work is "done." The tests it wrote pass.
3. The acceptance test runs and fails on its last line, with 0 items in the cart. The address saved, but the cart emptied.
4. The developer sends the failing scenario to the agent, which edits the code. The acceptance test passes on the next run.
5. The product owner places a trial order on the staging store with 3 items and a changed address. This is the user acceptance step.
6. The owner accepts the change, and it ships in the next release.

The acceptance test checked the rule the owner stated, which the agent's own tests never checked. This example is simplified. A real release would need more criteria, e.g. one for an empty address field.

What changes when a coding agent writes the code?

A coding agent can run an acceptance test itself and edit the code until the test passes. Under ATDD, the failing test becomes the agent's target. A pass then shows that the code fits the scenario, not that it meets the whole need.

Code written to pass one scenario can fit that scenario's data, e.g. by handling only the values the test uses. The agent can also edit the test instead of the code. If the agent writes the acceptance tests as well, they check its own reading of the request, so they verify the code without validating it. In the Acme example, the owner's trial order used 3 items, not the scenario's 2, so it tested the need with different data.

A practical adjustment is to treat the agent's passing run as evidence for the owner, not as the acceptance itself. People approve the criteria before the agent starts, as spec-driven development does, and the test files stay out of the agent's edits. The team's definition of done requires a passing run on the final code. The owner then tries the change with data the scenario did not use.

Who runs acceptance tests?

The Foundation Level syllabus says that the intended users should ideally perform acceptance testing. In practice, the answer depends on who must accept the software. The main types of acceptance testing differ in who takes part:

• User acceptance testing. Intended users, or a product owner acting for them, check that the software supports their tasks. UAT is often manual.
• Operational acceptance testing. The team that will run the system checks that it can keep the system running, e.g. by restoring data from a backup. It is also called production acceptance testing.
• Contractual and regulatory acceptance testing. Users or independent testers check custom software against the acceptance criteria in its contract. For regulated software, they check it against the regulations that apply, sometimes with a regulator witnessing or auditing the results.
• Alpha and beta testing. People outside the development organization try the software. Alpha testing takes place in the developer's test environment, and beta testing takes place at the testers' own sites, often after an alpha test.

Testers who do QA testing often help the owners write criteria and prepare test data, and developers automate the tests. The decision to accept stays with the people who must accept the software, because the tests check their needs.

What are the limits of acceptance testing?

Acceptance testing gives owners evidence for a decision, not proof that the software is correct. It has these limits:

• It checks only the criteria someone wrote. A need that nobody stated, e.g. an address the store cannot deliver to, has no test. No findings is not the same as complete coverage.
• It comes late. A misunderstood request that surfaces at acceptance means rework on a finished build. ATDD turns the criteria into tests before the code for this reason.
• Manual checks are hard to repeat. An owner who clicks through a change tries a few paths, usually the expected ones, and may not try them the same way next time.
• A pass can be a formality. An owner who approves a build without trying it adds no evidence beyond the team's own tests.

How is acceptance testing different from system testing?

The ISTQB defines system testing as a test level that focuses on verifying that a whole system meets its specified requirements. The development or test team usually runs it. Acceptance testing checks the same system against the needs of the people who will accept it, and those people take part or decide.

System testing is mostly verification, and acceptance testing is mostly validation. Both often follow whole user tasks from start to finish, and some teams reuse system tests as automated acceptance tests. Software can pass system testing and still fail acceptance when its specification left part of the need out.

FAQs

Can acceptance tests be automated?

Acceptance tests can be automated when each criterion states a result that a test can check, e.g. that the cart still holds its items. Automated acceptance tests can then run on each change. The decision to accept still belongs to people, and user acceptance testing is often manual.

What is acceptance test-driven development?

Acceptance test-driven development (ATDD) is a practice in which the team writes acceptance tests from the criteria before it writes the code. The tests use the language of the people who asked. They fail at first and pass once the code meets the criteria.

When does acceptance testing happen?

Acceptance testing usually happens last among the test levels, after system testing and before release. Agile teams often check each user story against its criteria when the story is built. Under ATDD, the tests are written before the code but still run on the finished change.

What is operational acceptance testing?

Operational acceptance testing is a form of acceptance testing in which the team that will run the system checks that it can keep the system running. A typical check restores data from a backup. It is also called production acceptance testing.