Glossary

Acceptance criteria are the conditions that a change must meet for the person who asked for it to accept it. They are written so that a person or a test can check each one.

Acceptance testing is a level of testing that checks whether a system meets its acceptance criteria and its users' needs. It lets the system's users and owners decide whether to accept it.

The accessibility tree is the browser's structured view of a page's elements, names, and roles, which assistive technology and many browser agents read the page through.

Adversarial testing is a testing approach that deliberately tries to break software with unexpected, hostile, or careless inputs and actions, instead of confirming the expected path.

An agent harness is the software around a language model that gives it tools, context, memory, and a loop, which turns the model into a working agent.

An agent hook is a script that a coding agent's harness runs at a fixed point in its loop. A Stop hook runs when the agent tries to end its turn.

The agent loop is the cycle in which a model requests an action, the harness runs the tool, and the model reads the result. It ends when the model requests no tool, or when a step limit, a time limit, or a person stops it.

An agent trajectory is the ordered sequence of an agent's actions, tool calls, and observations while it works on a task, which is what a transcript records.

Agentic engineering is a way of building software that directs coding agents while keeping engineering practices, from written specs and tests to code review and verification.

Agentic testing is software testing that AI agents carry out themselves, choosing, running, and judging checks on the software, rather than only writing test scripts for later.

AGENTS.md is a Markdown file in a repository that gives coding agents project instructions, e.g. how to build, test, and change the code.

An AI agent is a program that uses a language model in a loop to choose actions, call tools, and read the results until it finishes a task.

AI code attribution is the practice that records which code an AI tool wrote and under which prompt, e.g. with commit trailers.

AI code review is a review practice that uses a language model to read a pull request or diff and comment on possible bugs, style problems, and risks.

AI code verification is the practice that checks what AI-generated software does, by reading it, running its tests, and running the software, rather than detecting who wrote it.

AI slop is AI-generated output that looks finished but is bloated, duplicated, or poorly tested because nobody checked it closely. In code it includes verbose logic and tests that assert almost nothing.

Approval fatigue is a failure in which people approve a coding agent's permission prompts without reading them, because the prompts are so frequent that they stop protecting anything.

Arrange-act-assert is a test layout that sets up the inputs first, then runs the code under test, and then checks the result, each in its own block.

An assertion is a statement in a test that checks a condition and fails the test when the condition is false.

An assertion-free test is a test that runs code but checks no result, so it passes whenever the code does not crash, whether the output is right or wrong.

A background coding agent is a coding agent that works on a task without a person watching and returns a branch or pull request when it finishes.

Behavior-driven development is a practice that describes how software should behave as plain-language Given-When-Then scenarios that the team agrees on and tools can run as tests.

A blameless postmortem is an incident review that works out how a failure happened and how to prevent a repeat, without blaming the individuals involved.

The blast radius of a change is the set of features, users, and systems that could break because of it. That set is often much wider than the files in the diff.

Branch coverage is a code coverage measure that counts which outcomes of each decision point, e.g. both sides of an if statement, the tests have run.

A browser test framework is an open-source library that drives a real browser from test code, so a test can load pages, act on them, and check the results.

A bug bash is a time-boxed session that gathers people from across a team to use the software together and report as many defects as they can find.

A catching test is a test that is generated for one code change and designed to fail if that change introduced a bug, then usually thrown away after review.

A characterization test is a test that records what existing code does now, right or wrong, so later changes can be checked against that recorded behavior.

The Chrome DevTools Protocol is an interface that lets a program drive and inspect a Chromium-based browser, and many automation tools and browser agents are built on it.

A CI pipeline is the ordered set of automated steps, e.g. build, test, and package, that runs each time code is pushed or merged.

CI/CD is a practice that merges code often and builds and tests each change in an automated pipeline, so the software is kept ready to release.

Code coverage is a test measure that reports the percentage of a program's code that runs during its tests, which shows what was executed but not what was checked.

Code review is a practice that has people or tools read a proposed code change before it merges, to catch defects, share knowledge, and keep the codebase consistent.

A coding agent is an AI system that edits files and runs commands in a loop to complete a programming task with little step-by-step direction.

A coding agent transcript is a saved record that holds a coding session's prompts, replies, tool calls, and results. A collection of saved transcripts is called AI coding session history.

A command-line interface is a way of using a program that takes typed commands, flags, and input in a terminal or script, and returns output, errors, and an exit code.

A commit is a saved snapshot of changes in a Git repository, recorded with a message, an author, and a link to the commit before it.

A container is a packaged process that runs with its own filesystem and limits but shares the host's kernel, which makes it lighter and less isolated than a virtual machine.

A context window is the amount of text, measured in tokens, that a language model can take into account at once, including instructions, files, and history.

Continuous delivery is a practice that keeps every change that passes the pipeline ready to release, so a person can release it to production at any time with one decision.

Continuous deployment is a practice that releases every change that passes the automated pipeline to production, with no manual approval step between the merge and the release.

Continuous integration is a practice that merges every developer's changes into the main branch often, with an automated build and test run on each merge.

Coverage-guided fuzzing is a fuzzing method that keeps the inputs which reach new code paths and mutates them further, so the fuzzer explores a program efficiently.

Cross-model verification is a checking setup that asks a different AI model to review work that another model produced, in the hope that their blind spots differ.

Dynamic application security testing (DAST) is a security practice that probes a running application from the outside for vulnerabilities. It is separate from functional testing.

Debugging is the process that finds why software does the wrong thing, by reproducing the failure, narrowing down its cause, changing the code, and confirming the fix.

A definition of done is a shared checklist that states what evidence a piece of work must have before a team counts it as finished.

Design for testability is a design approach that makes software easier to test, e.g. by giving a CLI clear exit codes and machine-readable output.

A dev container is a container defined in the repository that provides a ready-made development environment with its tools installed.

Dynamic analysis is a method that observes a program while it runs, e.g. through tests, to find bugs that only appear at runtime.

Egress control is a network control that limits which outside hosts a sandbox or agent can reach, usually with an allowlist or a proxy.

An end-to-end test is a test that follows a whole user journey through a running system, from the interface through services and data, the way a real user would.

Environment parity is the degree to which development, test, staging, and production environments match, so a check in one predicts behavior in another.

An ephemeral environment is a short-lived copy of an application that is created for one change or test run and destroyed afterwards. A preview environment is one kind.

An equivalent mutant is a code change made during mutation testing that behaves exactly the same as the original program, so no test can ever detect or kill it.

An eval, short for evaluation, is a repeatable test of an AI system that runs a fixed set of tasks and scores the results, so versions of the system can be compared.

An exit code is the number that a program returns when it finishes, where 0 usually means success and any other value signals an error that scripts can check.

Exploratory testing is a testing approach that designs and runs tests at the same time, guided by what the tester learns about the software while using it.

Fail-open is a behavior that lets an action through when a check errors or times out, while fail-closed code denies it. Fail-open authentication is a common security bug.

A false completion claim is a coding agent's report that a task is "done," or that tests pass, when the work has not been built, run, or checked.

A false negative is a test result that misses a defect that does exist, so the check passes even though the software is broken.

A false pass is a test or check result that reports success while the software is broken, because the check missed the defect or lost the failure.

A false positive is a check result that reports a problem that does not exist, which wastes time and teaches people to ignore real findings.

Fault seeding is a technique that deliberately inserts known bugs into code to see how many of them a test suite or review process detects.

A flaky test is a test that both passes and fails on the same code. A single result from it cannot be trusted as evidence of a bug or of a fix.

Formal verification is a method that proves mathematically that a program or model meets a specification, instead of testing it on sample inputs.

Fuzzing is a testing technique that feeds a program large numbers of random or malformed inputs to find crashes, hangs, and bugs in input handling.

A ghost feature is a feature that exists in the code and may pass its tests but never runs for users, e.g. a handler that nothing calls.

Git bisect is a Git command that runs a binary search through a repository's history to find the first commit where a behavior changed.

A Git worktree is an extra working directory attached to the same repository, so several branches can be checked out at once, e.g. one per parallel agent.

Given-when-then is a scenario format that states a starting state, an action, and an expected outcome in plain language, used in behavior-driven development and acceptance criteria.

A golden file is a saved, approved copy of a program's output that a test compares new output against, failing when anything in the output changes.

Goodhart's law is the observation that a measure which becomes a target stops being a good measure, because people and systems optimize the number instead of the goal.

The happy path is the expected route through a feature, with valid input and no errors, that most demos and many generated tests follow.

A headless browser is a web browser that runs without a visible window and is controlled by code, so tests and agents can load pages and act on them.

A heisenbug is a bug that disappears or changes when someone tries to observe it, e.g. when a debugger or extra logging changes the timing.

A hermetic test is a test that depends only on what it declares and sets up itself, with no shared state, network, or leftover files. It gives the same result anywhere.

A hook is a script that a tool runs automatically at a set point in its workflow, e.g. Git running a check before it records a commit.

Human-in-the-loop is a way of running an AI system in which a person approves, corrects, or reviews its work at set points before the work takes effect.

A hydration error is a web app failure that happens when the HTML a server rendered does not match what the browser's JavaScript renders on load.

Idempotency is a property of an operation that gives the same result whether it runs once or several times, e.g. a payment request that is safe to retry.

An implicit oracle is a general failure signal that marks a result as wrong without any expected value, e.g. a crash.

Indirect prompt injection is a prompt injection that arrives through content an agent reads during a task, e.g. a web page, instead of from the user.

An integration test is a test that checks that separate parts of a system work together, e.g. a service and its database, rather than each part alone.

An invariant is a condition that must stay true for every valid state or input of a program, e.g. an order total that is never negative.

A jailbreak is a prompt that gets a language model to break its own safety rules. Prompt injection differs because it attacks an application built on the model, by mixing untrusted text into its instructions.

A large language model is a model that is trained on large amounts of text to generate text, including code, from a prompt. Coding agents use one to choose what to do next.

Least privilege is a security principle that gives a person, program, or agent only the access its task needs, for no longer than it needs it.

The lethal trifecta is a risk pattern in which an agent has access to private data, reads untrusted content, and can send data out, which lets prompt injection leak data.

Linting is a static check that compares source code against style and correctness rules with a tool called a linter, without running the code.

LLM-as-a-judge is an evaluation setup that uses a language model to grade output against written criteria, instead of or alongside deterministic checks.

Load testing is a performance test that runs software under the traffic expected in production, to measure response times and find where it slows down.

An MCP server is a program that exposes tools or data to AI agents over the Model Context Protocol, so an agent can call them in a standard way.

A merge queue is a system that tests each approved pull request against the current main branch and the changes ahead of it, and merges only those that pass.

Metamorphic testing is a technique that checks how a program's output should change when its input changes in a known way, so code can be tested without an exact expected output.

A microVM is a small virtual machine that starts quickly and gives each workload its own kernel, which isolates it more strongly than a container.

A minimal reproducible example is the smallest complete code, input, and setup that still shows a bug, so anyone can run it and see the same failure.

The Model Context Protocol is an open standard that lets AI applications connect to tools and data through MCP servers, so an agent can call them in a uniform way.

Monkey testing is a testing technique that feeds software random clicks, keystrokes, or inputs without a script, to see whether it crashes or misbehaves.

A mutant is a copy of a program with one small deliberate change, e.g. a flipped comparison, that mutation testing uses to check whether the tests fail.

Mutation score is a test-strength measure that reports the share of valid mutants a test suite detects, which shows how well the tests check behavior.

Mutation testing is a technique that checks a test suite by injecting small deliberate bugs, called mutants, and counting how many of them the tests catch.

Negative testing is a testing technique that checks how software handles invalid, unexpected, or missing input, instead of only the input it was designed for.

Nondeterminism is a property of a program or test that can produce different results from the same code and input, e.g. because of timing.

Observed vs. expected is the part of a bug report that states what the software did next to what it should have done, as facts rather than guesses about causes.

An out-of-scope edit is a change that a coding agent makes outside what its task asked for, e.g. an unrequested refactor.

The OWASP Top 10 is a list that the Open Worldwide Application Security Project publishes of the most critical security risk categories for web applications.

Package hallucination is a code generation failure that imports or installs a package that does not exist. An attacker can register the invented name and publish harmful code under it.

A partial oracle is a test oracle that checks some properties of a result when the full expected result is unknown, e.g. that an order total is never negative.

A pass on retry is a test result that fails first and then passes when the same test runs again on the same code. It marks the test as flaky or the bug as intermittent.

Playwright MCP is an open-source server that lets an AI agent control a real browser through the Model Context Protocol, using structured page snapshots instead of screenshots.

A pre-commit hook is a script that Git runs before it records a commit. It can lint, format, or test the staged changes and stop the commit if a check fails.

A pre-push hook is a script that Git runs before it sends commits to a remote repository, which can run slower checks than a pre-commit hook and stop the push.

Prompt injection is an attack that hides instructions in content an AI model reads, e.g. a web page, so the model follows them instead of its user.

Property-based testing is a technique that states rules that should hold for every input, then checks them against many generated inputs and shrinks any failing input.

A pull request is a request that asks to merge a set of commits into another branch, where people and tools review and check the change first.

Quality assurance is a set of activities that aims to prevent defects by improving how a team builds and tests software. Testing, a form of quality control, checks the product itself.

A quality gate is a point in a delivery pipeline where a change is measured against set criteria before it goes further. Some gates block the change, and others only report.

A race condition is a bug that makes a result depend on the timing or order of events running at the same time, e.g. two requests changing the same record.

Red teaming is a practice that has a team act as an attacker or hostile user to find weaknesses in a system before real attackers or users do.

Red-green-refactor is the test-driven development cycle that writes a failing test, then the code that makes it pass, then cleans up the code while the test stays green.

Refactoring is a practice that changes the structure of existing code without changing its behavior, e.g. to remove duplication, so the code is easier to read and change.

Regression testing is a type of testing that checks that a change has not broken behavior that worked before, including parts of the software the change did not touch.

A reproduction script is a runnable script that replays the steps that trigger a bug, so a person or an agent can reproduce the failure without guessing.

Reproduction steps are the exact actions, inputs, and conditions that make a bug happen again, written so that a person or a coding agent can follow them.

Retesting is a check that repeats the exact steps that exposed a defect on the fixed code, to confirm that the failure no longer happens. It is often called fix verification.

The review bottleneck is the queue that forms when code is produced faster than people can review it, so pull requests wait or get approved with less scrutiny.

Reward hacking is a behavior in which a coding agent satisfies the check it is graded on, e.g. the tests, without doing the task the user intended.

Root cause analysis is a method that traces a failure past its symptoms to the underlying cause, so the fix removes the reason the failure happened.

Row-level security is a database feature that limits which rows each user can read or change, so one user's data stays hidden from another.

A rubber-stamp review is a code review that approves a change without checking it closely, often because the change is large or the queue is long.

A rules file backdoor is an attack that hides instructions in a coding agent's instruction or rules file. The agent then writes harmful code while appearing to follow the project's rules.

Runtime verification is a method that checks a running program against stated properties by observing its behavior, usually through a monitor that reads its execution. Developers also use the term for running the software to check a change.

A sandbox is an isolated environment that limits which files and network hosts the code inside it can reach, so a bad command can damage only what the sandbox allows.

Sanity testing is a quick, narrow check that a specific fix or change behaves sensibly before deeper testing, often used interchangeably with smoke testing.

Static application security testing (SAST) is a security practice that scans source code for vulnerabilities without running it. It is separate from functional testing.

Secret scanning is a check that searches code and history for credentials, e.g. API keys, so they can be removed and rotated before someone misuses them.

A self-healing test is a test that updates its own locators or steps when the app changes, which cuts upkeep but can also hide real breakage.

Shift left is an approach that moves testing and other quality checks earlier in development, when a problem is smaller and cheaper to fix.

Shrinking is a step in property-based testing that reduces a failing generated input to the smallest input it can find that still fails, so the bug is easier to understand.

SIGPIPE is the signal that a program receives when it writes to a pipe whose reader has closed. By default it ends the program, and shells report exit code 141.

A skipped test is a test that is marked not to run, which many tools still report as part of a passing suite, so it can hide a missing check.

Slopsquatting is an attack that registers package names that language models tend to invent, so code that installs a hallucinated package installs the attacker's code instead.

A smoke test is a quick, shallow test that a build starts and its most basic functions work, run before deeper testing is worth the time.

A snapshot test is a test that saves a program's output once and fails when later output differs, until someone reviews and approves the new output.

A software regression is a defect in which something that worked before a change stops working after it, often in a part of the code the change did not touch.

A software specification is a written description that states what a program should do, its inputs and outputs, and its constraints, so people and tests can check the result.

Software testing is the practice that runs or examines a program to find differences between what it does and what it should do, before users find them.

Spec drift is the gap that opens when code and its specification change separately, so the written spec no longer describes what the software does.

Spec-driven development is a way of working that writes a specification of what software should do before a coding agent writes it. The spec is then used to guide and check the work.

Specification gaming is a behavior in which an AI system satisfies the literal objective it was given without doing the task the objective was meant to capture.

A stack trace is a report that lists the chain of function calls active when an error happened, from the failing line back through the calls that led to it.

A staging environment is a production-like copy of an application that teams use for final testing before release, running the same build with similar settings but no real users.

The standard streams are the three channels that every process gets by default, which are standard input for data in, standard output for results, and standard error for diagnostics.

Static analysis is a method that examines source code without running it, using rules and type information to flag likely bugs and security flaws.

A status code is a number that a web server returns with each HTTP response to report the result, e.g. 404 for not found.

stderr is the standard error stream that a program writes errors and diagnostics to, so they stay separate from the results on standard output.

stdout is the standard output stream that a program writes its normal results to, so another program, a file, or a terminal can read them.

A subagent is a coding agent that another agent starts to handle one part of a task in its own context, then returns a result to the parent.

System testing is a level of testing that checks the complete, integrated system against its requirements, after integration testing and before acceptance testing.

The system under test is the part of the software that a test exercises and checks, as opposed to the harness around it, including its test doubles.

A tautological test is a test that cannot fail because its expected value comes from the same code, mock, or constant as the result it checks. It passes whether the code is right or wrong.

Technical debt is the future cost of shortcuts in code or design, which a team pays as slower changes and more bugs until someone cleans the shortcuts up.

Test automation is the use of software to run tests and compare actual results with expected results, so the same checks can repeat without a person doing each step.

A test charter is a short mission statement for an exploratory testing session that names the area to explore, the risks to look for, and a time box.

Test data management is the practice that creates, refreshes, and cleans up the data tests need, so each run starts from a known state without real personal data.

A test double is a stand-in for a real component that a test uses in its place. Stubs return canned answers, and mocks also check that they were called as expected.

A test fixture is the fixed setup that a test needs before it runs, e.g. seeded data, together with the steps that clean it up.

A test harness is the set of code and tools that runs tests against a system, including drivers that start it, test doubles for its dependencies, and checks on the results.

Test impact analysis is a technique that maps which tests exercise which code, so a change runs only the tests it could affect instead of the whole suite.

Test isolation is a property of a test that runs with its own state and data, so its result does not depend on other tests, their order, or leftover files.

A test oracle is the source of truth that a test compares a result against, e.g. a specification that states the correct output for a given input.

The test oracle problem is the difficulty of deciding whether a test passed or failed for a given input and state, which is hardest when nobody knows the correct answer.

The test pyramid is a model of a test suite that has many fast unit tests at the base and fewer integration tests in the middle. A few end-to-end tests sit at the top.

Test quarantine is a policy that moves a flaky test out of the blocking suite while it is fixed. The test still runs and reports but no longer fails the build.

A test suite is a collection of tests that a project runs together to check its code, e.g. every unit test in a repository.

Test tampering is a coding agent behavior that edits, skips, or deletes a failing test so the suite passes, instead of fixing the code the test was checking.

Test-driven development is a practice that writes a failing test before the code, then only enough code to pass it, then refactors, in small repeated cycles.

Test-order dependence is a flaw that makes a test pass or fail depending on which tests ran before it, usually because they share state.

Testing in production is a practice that checks software after release with real traffic, e.g. through monitoring, alongside testing before release.

Tool calling is a model capability that lets a language model request a named function with arguments, which the surrounding harness runs and returns results from.

Tool-call spoofing is an agent failure in which a transcript shows a tool call or tool output that did not happen, so the record no longer matches what ran.

A TTY is a terminal device that a program can detect, and many programs change their output, color, or prompts when they write to a pipe instead of a TTY.

A two-account test is a behavior check that signs in as two different users and confirms that neither can see or change the other's data.

Type checking is a static check that verifies values are used according to their declared types, e.g. that a function expecting a number never receives a string.

A unit test is a test that checks the smallest testable part of a program, e.g. one function, in isolation from the rest of the system.

A user journey test is a test that follows one complete task a user performs, e.g. signing up, from start to finish through the running software, and checks each step.

Verification debt is the gap that grows between how much AI-written code a team produces and how much of it anyone has checked by running or testing it.

Vibe coding is a way of building software in which a person describes the result to an AI and accepts the code without reading it, judging it only by using the app.