What is test data seeding?

What is test data seeding?

Test data seeding is a setup step that loads known records into an empty test database before tests run. It is a form of database seeding. The records come from a seed script in the repository. Each run then starts from the same state, so the next run can reproduce a failure that the code causes.

The International Software Testing Qualifications Board (ISTQB) defines test data as the data needed to run tests. Seeding is one part of test data management. Test data management is the practice that creates, refreshes, and cleans up the data tests need, so each run starts from a known state without real personal data. Its other parts include masking production data, e.g. replacing real email addresses, and generating synthetic records.

Seeding is most common in integration testing and end-to-end testing, because both often read and write a real database. It runs wherever those tests run, e.g. a fresh sandbox where a coding agent works. Without a seed and a reset, tests read whatever rows earlier runs left behind, so results can change between runs.

How does test data seeding work?

A seeded test run usually has five steps:

1. The pipeline or test runner creates an empty database for this run.
2. A migration tool builds the tables from the current schema.
3. A seed script inserts the known records.
4. The tests run against those records, and each test adds only the rows it needs.
5. After each test, the test runner restores the seeded state, e.g. by rolling back the test's transaction if the app uses the test's connection.

Seed data usually comes in three kinds:

• Reference data. The app needs these records in every environment, e.g. the list of shipping zones.
• Scenario data. These records set up the cases that tests check, e.g. the customer that a user journey test signs in with. They belong in a seed that production never runs.
• Generated data. A library makes up many records to cover more cases, e.g. 500 customers. The records repeat on each run when the generator gets a fixed current date and a fixed random seed, the number that sets its sequence.

In Ruby on Rails, bin/rails db:seed runs db/seeds.rb, and bin/rails db:setup creates the database, loads the schema, and seeds it. The Rails migrations guide says seed code "should be idempotent so that it can be executed at any point in every environment." Idempotency means running the seed twice leaves the same records as running it once.

Steps 1 to 3 also run when a pipeline starts an ephemeral environment for one change, so reviewers see known records.

What is an example of test data seeding?

Here is an illustrative example. Acme Co. sells furniture online. A developer asks a coding agent to "Let customers edit their delivery address during checkout." Acme's end-to-end test for checkout signs in as a customer that this seed script creates:

// scripts/seed.ts runs after the migrations, on an empty test database
const shopper = await db.customers.upsert({
  where: { email: "shopper@example.com" },
  create: { email: "shopper@example.com" },
  update: {},
});
await db.cartItems.deleteMany({ where: { customerId: shopper.id } });
await db.cartItems.create({
  data: { customerId: shopper.id, sku: "CHAIR-01", quantity: 2, price: "120.00" },
});

The script is idempotent, so it can rerun on a local database that already holds the customer. The upsert call creates the customer only once, and the cart is emptied before the 2 chairs go in. The checkout test reads those records:

test("editing the address keeps the cart", async ({ page }) => {
  await signIn(page, "shopper@example.com");
  await page.goto("https://shop.example.com/checkout");
  await page.getByLabel("Delivery address").fill("12 Elm Street");
  await page.getByRole("button", { name: "Save address" }).click();
  await expect(page.getByTestId("cart-count")).toHaveText("2");
  await expect(page.getByTestId("cart-total")).toHaveText("$240.00");
});

The change goes through five steps:

1. The pipeline creates an empty database, runs the migrations, and runs npm run db:seed.
2. The test signs in as shopper@example.com, whose cart holds 2 chairs at $120.00 each.
3. The test saves 12 Elm Street as the delivery address and fails. The address saved, but the cart emptied.
4. The agent fixes the checkout code, and the pipeline seeds a fresh database and runs the test again.
5. The cart still holds 2 items with a total of $240.00, and the test passes.

The seed put the same 2 chairs in the same cart on each run, so the failure repeated until the fix. The passing run used the same records as the failing one. This example is simplified. A real seed would hold more scenarios, e.g. a customer with no saved address.

What changes when a coding agent writes the code?

A coding agent that edits the code can also edit the seed, and a changed seed can make a failing test pass while the code stays wrong. Suppose Acme's seed held a customer with no saved address, and the agent's checkout code failed for that customer. If the agent deleted that record, the suite would pass, and real customers with no saved address would still hit the bug.

Generated seed values can also look plausible and still be wrong. In a sample data tutorial, Cursor generated records for 10 classic cars, and one car's years of manufacture were wrong. A test that takes its expected value from the same seed passes on the wrong value.

A practical adjustment is to review a deleted or changed seed record as closely as a deleted test. Keep the records for odd cases in a seed file that the agent's task does not edit. Check generated values against a real source before a test relies on them.

What are the limits of seeded data?

Seeded data makes runs repeatable, but it has these limits:

• It holds only the cases someone wrote. Production builds up odd records, e.g. accounts created before a field was required, and a seed rarely has them. A staging environment with masked production data holds more of them.
• It drifts from the schema. A seed written for an older schema breaks when a migration renames a column it writes. Running the seed in the pipeline on a database built from the current migrations turns a broken seed into a failed build. Editing the seed in the same change as the migration keeps the two in step.
• Shared records couple tests. When one test changes a seeded record that another reads, the second test's result depends on test order. That can make it a flaky test, and test isolation fixes it by having each test reset what it changes.
• It costs time. A large seed slows each run, so teams often seed once and reset between tests.

A passing run on seeded data shows only that the code handled those records.

How is seeding different from fixtures and snapshots?

Seeding, fixtures, and database snapshots each put tests in a known state, at different scopes:

A test fixture often starts from seeded records and adds the rows its own test changes. Rails and Django also call files of records that load before tests "fixtures," which is a form of seeding. A factory, e.g. factory_bot, builds records inside each test. In PostgreSQL, CREATE DATABASE can copy a seeded database named as its template while no other session is connected to it, a form of database branching. A snapshot goes stale when the schema changes, so teams rebuild it from the seed.

How do you give each run the data it needs?

Start each run from a clean checkout and an empty database, apply the migrations, and run the seed before the tests start. A fresh database for each run keeps one run's writes out of another's data. Then let each test create or reset only the rows it changes.

Seeded tests check only the workflows that someone wrote tests for. RunStory runs your software in a separate environment, tries relevant workflows, and checks the results. You keep working. It is in private alpha for CLIs and web apps.

Join the RunStory alpha →

FAQs

Where should seed scripts live in a project?

Seed scripts usually live in the repository next to the database migrations, as the Rails seeds file does. Records that only tests use belong in a separate seed that production never runs, while reference data can share one seed across environments.

Should test data be random or fixed?

Test data in a seed is usually fixed, so a rerun uses the same records and a failure can be repeated. Generated data can cover more cases, and it stays repeatable when its generator starts from a fixed random seed and a fixed current date.

How do you keep seed data in step with the schema?

Seed data stays in step with the schema when each change that alters a table also updates the seed. Running the seed in the pipeline on freshly migrated tables turns a forgotten update into a failed build.

Can a coding agent write seed data?

A coding agent can write seed data, and its seed changes need the same review as its test changes. An agent can delete an awkward record to make a failing test pass, and a test that takes its expected value from a wrong generated value still passes.