What is a quality gate in CI/CD?

What is a quality gate in CI/CD?

A quality gate is a point in a delivery pipeline where a change is measured against set criteria before it goes further. The result is a pass or a fail. In a continuous integration and delivery (CI/CD) pipeline, the criteria are usually automated checks, e.g. passing tests. Some gates stop a failing change, and others only report it.

The International Software Testing Qualifications Board (ISTQB) defines a quality gate as a milestone where the decision to move to the next phase rests on predefined quality criteria. That sense covers any stage, from a merge to a production release. Some code quality tools use the term more narrowly, for thresholds on static analysis and coverage results in each branch or pull request.

A gate turns a team's standard into a check that runs on each change. The standard then holds even when nobody remembers to apply it.

How does a quality gate work?

A quality gate runs at a fixed point in the pipeline, most often before a pull request merges. A gate before merge works in six steps:

1. A developer or a coding agent pushes a change to a pull request, and the CI service starts a run.
2. The run executes each check that the gate names, e.g. the unit tests.
3. Each check reports a pass, a fail, or a measured value, e.g. the share of changed lines that the tests ran.
4. The gate compares each result with its criterion, e.g. "no failing tests."
5. If every criterion holds, the gate reports a pass, and the change can move to the next stage.
6. If any criterion fails, the gate reports a fail and names the check that failed.

A merge gate usually combines criteria of these kinds:

• Tests. No unit test or end-to-end test fails.
• Coverage. Code coverage does not drop compared with the main branch.
• Scans. Static analysis and dependency scans add no finding above a set severity.
• Review. A person other than the author approves the pull request.

A blocking gate keeps a failed change where it is until a later run passes. A reporting gate shows the same result, and a person chooses whether to go on. Choosing a mode for each check is the trade between blocking and non-blocking checks.

A second gate often runs before release, with criteria of its own, e.g. a person's approval of the deployment.

What is an example of a quality gate?

Here is an illustrative example. Acme Co. sells furniture online. Its team requires one check, named gate, to pass before a pull request can merge into main. The gate job in its GitHub Actions workflow runs after the build, unit test, and end-to-end test jobs:

jobs:
  gate:
    needs: [build, unit, e2e]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Fail unless the 3 jobs passed
        if: >-
          needs.build.result != 'success' ||
          needs.unit.result != 'success' ||
          needs.e2e.result != 'success'
        run: exit 1

The if: always() line makes the gate run even when a job it needs has failed. A developer at Acme asks a coding agent to "Let customers edit their delivery address during checkout." The change then meets the gate:

1. The agent changes the checkout code and opens a pull request.
2. The build passes, and the unit test job reports 48 tests passed.
3. The end-to-end test fails with expected cart to have 2 items, received 0. The address saved, but the cart emptied.
4. The gate job reads failure for e2e, exits with code 1, and the pull request cannot merge.
5. The developer pastes the failing line into the agent's session, and the agent fixes the code that cleared the cart.
6. The 3 jobs pass on the next commit, so gate passes.
7. A reviewer reads the diff, approves it, and merges the change.

This example is simplified. A real gate might also require the branch to be up to date with main, so the checks cover changes that merged after the last run.

What changes when a coding agent writes the code?

A coding agent's output can say "done" before any gate has run. The gate's result is then often the first evidence from outside the agent's session about whether the change meets the team's definition of done.

An agent told to make CI pass can work toward the gate's criteria instead of the request. It can meet them without fixing the code, e.g. by skipping a failing test. The files that define the checks, e.g. a test runner's configuration, often live in the repository, so the agent can lower a threshold in the same pull request.

GitHub's documentation states that any person or integration with write access can set the state of a status check. An agent with the developer's access can therefore post a passing status under the gate's name.

To keep the gate outside the agent's reach, a team can require a code owner's review for changes to workflow files and test configuration. Each required check can name the app that must report it. The gate can also fail a pull request that deletes a test or marks one as skipped. A reviewer of agent pull requests can read changes to tests and CI files first.

What makes a good quality gate?

A gate is trustworthy when a pass and a fail both mean something about the change. Good gates share these traits:

• Each criterion is objective. A machine can check it without judgment, e.g. "the build compiles."
• Each failure is real. Two runs on the same code give the same result. A flaky test fails the gate at random, and people learn to rerun it until it passes. Many teams quarantine flaky tests, which moves them out of the gate until they are fixed.
• It is fast enough to wait for. Agents open many pull requests, so a slow gate becomes a queue. Fast checks can also run earlier, on the developer's machine or in the agent's loop, which is shift-left testing.
• It checks behavior, not only a number. At least one test of a real workflow, e.g. checkout, keeps the gate tied to what users do.
• A named owner controls it. A person or group other than the change's author approves changes to the criteria and to the files that define them. The owner also reviews a gate that fails often or catches little.

What are the limits of quality gates?

Gates have these limits:

• A gate checks only what its criteria name. A change can pass each check and still build the wrong feature. A run with gaps does not establish that the whole app works.
• A gate can pass without running a check. GitHub counts a required check as passed when its status is successful, skipped, or neutral. Without if: always(), a failed e2e job would skip Acme's gate job, and the pull request could merge.
• A number can rise without better tests. A coverage criterion counts lines that ran, not results that were checked. A mutation score shows whether the tests detect injected bugs.
• Admins can bypass it. By default, a GitHub branch protection rule does not apply to repository admins.
• Judgment does not fit a threshold. Whether a change does what was asked still needs a person, whose approval is a human-in-the-loop step.

How is a quality gate different from a required status check?

A quality gate names the criteria a change must meet and the stage where they apply. A required status check is one way a code host enforces a gate before merge. On GitHub, a branch protection rule or a ruleset lists the checks that must pass before a pull request can merge into a protected branch. One gate can combine several required checks with a required review, and a release gate may use no status check at all.

FAQs

Who should own a quality gate?

A quality gate should belong to the team that owns the code, with a named person or group who approves changes to its criteria. Neither the developer nor the coding agent that wrote a change should be able to loosen the gate in that same change.

How do teams enforce quality gates automatically?

Teams enforce quality gates automatically by running the gate's checks in CI and making them required before merge, e.g. with a GitHub branch protection rule. The pull request then cannot merge until each required check passes, unless an admin bypasses the rule.

When should a build be marked failed?

A build should be marked failed when a criterion the team agreed on is not met, e.g. a failing test. A result that nobody has agreed to act on should only report, because a gate that fails for no real reason teaches people to rerun it until it passes.

Should a gate block on code coverage?

A gate can block on code coverage when the criterion is no drop compared with the main branch. Coverage counts the lines that ran, not the results that were checked, so a coverage gate should run next to a mutation score or a review of the tests.