What is smoke testing?

What is smoke testing?

Smoke testing is a short set of checks that shows whether a build starts and its core functions work before deeper testing begins. Some teams call it build verification testing or confidence testing. Others call it sanity testing, though that term often means a narrower check. When a smoke test fails, the team usually rejects the build.

The International Software Testing Qualifications Board (ISTQB) defines smoke testing as a test type used to gain enough confidence that a test object is ready for planned testing. The test object is the software being tested.

Smoke testing is one part of software testing. It runs first because a few fast checks can stop a broken build before slower end-to-end tests run on it. A person can run the checks by hand, but teams usually script them.

How does smoke testing work?

A smoke test fits into a build pipeline in this order:

1. The continuous integration (CI) pipeline builds the software from the change.
2. The pipeline starts the software in a test environment.
3. The smoke suite runs a few fast checks against it.
4. The suite ends with an exit code, and a nonzero code means the suite failed.
5. If every check passes, slower suites run next. If one fails, the pipeline stops and reports it.

A good smoke check covers these points:

• Startup. The software builds and starts without a crash.
• Main paths. Each main feature gets one quick check, e.g. adding a product to the cart.
• Clear failures. A check fails on a crash, an error status, or an error page.
• Speed. The suite finishes in minutes, so it can run on each build.
• Stable inputs. On a test server, fixed test data and no live outside services mean a failure points at the build.

For a command-line application, a smoke check might run acme --version and one real export. For a web app, a smoke check can load the main pages in a headless browser and fail on a page error. Teams often mark the smoke checks and run only those, e.g. with pytest -m smoke.

The suite is short enough to run in a pre-push hook. Many teams run it again after each deploy to a staging environment and to production, because a deploy can break startup when the code did not change.

What is an example of a smoke test?

Here is an illustrative example. Acme Co. sells furniture online. A developer asks a coding agent to "Let customers edit their delivery address during checkout." The agent edits the checkout code, runs the unit tests, and reports that the task is "done."

The CI pipeline builds the web store, starts it on a test server, and runs this smoke script:

#!/usr/bin/env bash
set -euo pipefail
BASE="https://test.shop.example.com"
curl -fsS "$BASE/health" > /dev/null
curl -fsS "$BASE/products/oak-desk" > /dev/null
curl -fsS "$BASE/checkout" > /dev/null
echo "smoke: 3 checks passed"

The -f flag makes curl exit with code 22 when the server returns an HTTP status of 400 or above. The -e option in set -euo pipefail stops the script at the first failed command. The build succeeds and the first two checks pass, but the third one fails:

curl: (22) The requested URL returned error: 500

The pipeline stops before the end-to-end suite starts. The change made the checkout page read an address lookup service URL from ADDRESS_API_URL. The test server does not set that variable, so the page returns an error. The unit tests passed because they never loaded the checkout page.

The developer adds the variable to the settings for the test server, staging, and production, and the smoke test passes on the next build. This example is simplified. A real smoke suite would also load checkout in a headless browser, because a page can return 200 and still fail to render.

What changes when a coding agent writes the code?

A coding agent can finish a task after running only the unit tests. Unit tests check one part at a time, so they can pass while the app fails to start or a main page returns an error. The agent's final message then says "done" about software that nobody ran, which is a false completion claim.

A smoke test closes that gap at low cost, because it starts the software the agent changed. If the agent starts the app itself, the run can still pass on settings that exist only in its session, e.g. an ADDRESS_API_URL value in a local file. The same build can fail on a clean test server.

An agent that is told to make a failing check pass can also edit the script instead of the code, e.g. by deleting the failing curl line. That edit is a form of test tampering.

A safer setup keeps the smoke script in a file the agent does not edit and runs it on a clean test server, outside the agent's session. The team then treats the smoke output as the evidence, not the agent's summary. The same rule holds for any check of AI-generated code.

What are the limits of smoke testing?

Smoke testing trades depth for speed, and that trade sets its limits:

• Shallow checks miss wrong results. Suppose the change in the illustrative example had a second bug. The address saved, but the cart emptied. The smoke script's checkout check only loads the page, so it would still pass.
• One path per feature. A smoke test follows the expected path and skips edge cases, e.g. an empty cart.
• A pass is weak evidence. A smoke run skips most of the app, so a pass does not establish that the whole app works.
• Unstable checks block good builds. A check that calls a live outside service can fail with no bug in the build. That makes it a flaky test, and teams start to ignore it.
• It does not replace regression testing. Regression testing checks that behavior that worked before still works across the app. That takes far more tests than a smoke suite holds.

How is smoke testing different from sanity testing?

Sanity testing is a quick, narrow check that one fix or change behaves sensibly before deeper testing. Smoke testing is broad and shallow across the whole build, while sanity testing is narrow and aimed at the part that changed. Many teams use the two words as synonyms. The ISTQB glossary has an entry for smoke testing but none for sanity testing. Teams that use both words should define each one.

The table adds regression testing, which is often compared with both:

What should run after a smoke test passes?

After a smoke test passes, run the checks that go deeper. Start with the workflows the change touched, then the regression suite and end-to-end tests of the main user journeys.

A smoke test shows only that the software starts and the checked paths respond, not that the workflows around a change still work. RunStory considers your prompts and code changes to decide what to test. It runs your software in a separate environment, tries relevant workflows, and checks the results. It is in private alpha for CLIs and web apps.

Join the RunStory alpha →

FAQs

When is smoke testing performed?

Smoke testing is performed first on each build, before any slower suite runs on it. Some teams also run it in a pre-push hook, and many repeat it after each deploy to staging and production.

Is a smoke test the same as a build check?

A smoke test goes further than a build check that only confirms the code compiles or packages. A smoke test also starts the software and checks its main paths. Some teams call a smoke suite a build verification test, and that name leads people to confuse the two.

Can a smoke test be automated?

A smoke test can be automated, and many teams run it as a script in the CI pipeline, e.g. a few HTTP requests against the running app. A person can run the same checks by hand, but teams usually script them so they can run on each build.

How is smoke testing different from regression testing?

Smoke testing checks that a build starts and its main paths respond, using a few fast checks. Regression testing checks that behavior that worked before still works across the app, which takes a much larger suite. Smoke tests usually run first, and regression tests run on builds that pass.