Debugging and reproducing failures
How to find, reproduce, and fix bugs, including flaky tests, race conditions, and regressions, and how to confirm that a fix worked.
Start here
What is debugging?
Debugging is finding and fixing the bug behind a software failure by reproducing it, narrowing down its cause, and confirming that the fix works.
Concepts
What are reproduction steps in a bug report?
Reproduction steps are the exact actions, inputs, and conditions needed to make a bug happen again, written so a person or an agent can follow them.
What is a flaky test?
A flaky test is one that both passes and fails on the same code, so a single result cannot be trusted as evidence of a bug or of a fix.
What is a minimal reproducible example?
A minimal reproducible example is the smallest self-contained program, input, or set of steps that still shows a bug, so anyone can run it.
What is git bisect?
Git bisect finds the commit that introduced a bug by binary search, testing commits between a known good one and a known bad one to halve the range.
What is root cause analysis (RCA)?
Root cause analysis is working back from a failure to the underlying reason it happened, so a fix removes the cause instead of hiding the symptom.
How-to guides
How to tell a flaky test from a real bug
Telling a flaky test from a real bug means rerunning the failure under controlled conditions and keeping the evidence, not retrying until it passes.
How to verify a bug fix
Verifying a bug fix means watching the steps that exposed a bug fail on the old code and pass on the fixed code, then checking that nothing nearby broke.
How to write a bug report a coding agent can fix
A bug report a coding agent can fix describes one failure in enough detail to rerun it, with the evidence as text and a command that fails before the fix.
Common questions
Why do AI coding agents break working features?
Coding agents break working features when edits reach beyond the task, fix symptoms in the wrong place, or change code the tests do not cover.
Failures and bugs
What is a race condition, and how do you reproduce one?
A race condition is a bug where the result depends on the timing or order of events, e.g. two requests changing the same data at the same moment.
Terms in this topic
- Blameless postmortem
- A blameless postmortem is an incident review that works out how a failure happened and how to prevent a repeat, without blaming the individuals involved.
- Debugging
- Debugging is the process that finds why software does the wrong thing, by reproducing the failure, narrowing down its cause, changing the code, and confirming the fix.
- Flaky test
- A flaky test is a test that both passes and fails on the same code. A single result from it cannot be trusted as evidence of a bug or of a fix.
- Git bisect
- Git bisect is a Git command that runs a binary search through a repository's history to find the first commit where a behavior changed.
- Heisenbug
- A heisenbug is a bug that disappears or changes when someone tries to observe it, e.g. when a debugger or extra logging changes the timing.
- Minimal reproducible example
- A minimal reproducible example is the smallest complete code, input, and setup that still shows a bug, so anyone can run it and see the same failure.
- Observed vs. expected
- Observed vs. expected is the part of a bug report that states what the software did next to what it should have done, as facts rather than guesses about causes.
- Pass on retry
- A pass on retry is a test result that fails first and then passes when the same test runs again on the same code. It marks the test as flaky or the bug as intermittent.
- Race condition
- A race condition is a bug that makes a result depend on the timing or order of events running at the same time, e.g. two requests changing the same record.
- Reproduction script
- A reproduction script is a runnable script that replays the steps that trigger a bug, so a person or an agent can reproduce the failure without guessing.
- Reproduction steps
- Reproduction steps are the exact actions, inputs, and conditions that make a bug happen again, written so that a person or a coding agent can follow them.
- Retesting
- Retesting is a check that repeats the exact steps that exposed a defect on the fixed code, to confirm that the failure no longer happens. It is often called fix verification.
- Root cause analysis (RCA)
- Root cause analysis is a method that traces a failure past its symptoms to the underlying cause, so the fix removes the reason the failure happened.
- Test quarantine
- Test quarantine is a policy that moves a flaky test out of the blocking suite while it is fixed. The test still runs and reports but no longer fails the build.