Testing in the AI Era: Using AI to Write Tests

A test whose expected value was produced by running the code under test can only confirm what the code already does; if the bug is already there, the test locks it in.

If you generate tests with an assistant, the pattern may be familiar: the PR adds forty tests, coverage ticks up two points, CI is green, and the same billing bug still reaches production on Thursday. The tests never had a way to disagree with the code.

This article covers the three shapes generated tests take when they pass without protecting anything, and the three checks that fix them: prompting from the requirement rather than the implementation, breaking the code on purpose, and reading the mocks before the assertions.

Why Does Coverage Rise While Bugs Still Ship?

Coverage measures execution, not verification. Jest’s coverage providers (Babel/Istanbul by default, V8 optional) and Vitest’s coverage providers (V8 by default, Istanbul optional) report the same four metrics: statements, branches, functions and lines. Neither evaluates whether any assertion could fail. A test that calls a function and asserts on whatever came back covers every line it touched, exactly as a test with a correct assertion would.

That is why the number keeps climbing while bugs keep shipping. Generated tests are cheap, so more code gets executed under test. But a test derived from the implementation shares every defect with the implementation, and the coverage report has no column for that.

Failure One: The Test Asserts What the Code Already Returns

A model given only the source has one oracle for expected values: the source. It reasons about (or runs) the function, observes the output, and writes that output as the literal in the assertion. If the function is wrong, the literal is wrong in the same way.

Take a discount function with a copy-paste bug:

// discount.ts
export function calculateDiscount(price: number, tier: "silver" | "gold"): number {
  const rate = tier === "gold" ? 0.25 : 0.25; // bug: silver should be 0.15
  return price * (1 - rate);
}

The describe/it/expect API below works in both Jest and Vitest; Jest exposes it globally, while Vitest requires an import or globals: true.

import { calculateDiscount } from "./discount";

// Generated from the implementation. 75 is what the buggy function returned.
it("applies silver discount", () => {
  expect(calculateDiscount(100, "silver")).toBe(75);
});

// Written from the rule: silver is 15% off, so 100 * 0.85.
it("charges 85 for a 100 silver order", () => {
  expect(calculateDiscount(100, "silver")).toBe(85);
});

The first test passes against the buggy function. The second fails, which is the point. In a diff, 75 and 85 look equally legitimate. Nothing in the syntax tells a reviewer that one was computed from the pricing rule and the other was copied from the function’s output. The only defence is asking where the number came from.

Failure Two: Generated Tests Only Cover the Happy Path

A generated suite tests what the prompt described, and a prompt that names a function describes only its normal operation. The cases a generated suite most often omits (malformed input, an unreachable dependency, a timeout) are the ones the prompt never mentioned, so the model had no reason to write them.

The result is five near-identical tests for well-formed input with valid tiers and none for a negative price, an unknown tier string, or an upstream service that never answers. Those are the paths that reach production untested, because they are also the paths developers exercise least by hand.

Failure Three: The Unit Under Test Is Mocked

If a test replaces the function it claims to test with a mock, its assertion verifies the mock’s configured return value rather than the function’s behaviour. Models mock aggressively because mocking makes tests pass reliably, and a test that mocks the database, the network client and the service under test will pass under any implementation.

The pattern is easier to see with an injected dependency, which keeps the example free of jest.mock versus vi.mock differences:

// cart.ts
import type { calculateDiscount } from "./discount";

export function cartTotal(price: number, tier: "silver" | "gold",
                          discount: typeof calculateDiscount): number {
  return Math.round(discount(price, tier) * 100) / 100;
}

// cart.test.ts
import { cartTotal } from "./cart";

// Before: the fake is the thing being checked.
it("returns discounted total", () => {
  const fake = () => 85;
  expect(cartTotal(100, "silver", fake)).toBe(85);
});

// After: assert on what cartTotal did, using a fake that exposes it.
it("rounds the discounted price to cents", () => {
  const fake = () => 85.004999;
  expect(cartTotal(100, "silver", fake)).toBe(85);
});

The “before” test would still pass if cartTotal ignored its arguments and returned 85. The “after” test hands the fake a value that only comes out right if the rounding logic runs, so it observes the function rather than the stub.

Fix One: Give AI Unit Test Generation the Requirement, Not the Code

Asking a model to write tests for a function name gives it only the implementation to work from. Supplying the business rules as written requirements gives it something the code cannot supply: a specification that can disagree with the code.

The weak prompt:

Write unit tests for calculateDiscount.

The stronger prompt:

Write unit tests for calculateDiscount in discount.ts.

Rules:
- Silver tier is 15% off the price.
- Gold tier is 25% off the price.
- Price must be non-negative; a negative price throws.
- The result is rounded to two decimal places.

Compute every expected value from these rules, not from the
current implementation. Include at least one invalid-input case
per rule. Name each test as the rule it checks.

The second prompt produces expect(calculateDiscount(100, "silver")).toBe(85) because 85 is what the rule says, and it fails against the buggy function on the first run. Test names that read as rules (“charges 85 for a 100 silver order”) also make the suite reviewable as a specification. The prompt does not guarantee the model ignores the source; it gives the model a source of truth that outranks it.

Fix Two: Break the Code and Watch for Red

The fastest check on a generated test suite is to change one constant or flip one comparison in the code under test and run the tests again. If everything stays green, the suite was not protecting that code.

1. Fix the bug in discount.ts (silver rate to 0.15), then change 0.15 to 0.05.
2. Run your test command.
3. Expected: "charges 85" fails with received 95.
   If nothing fails, no test asserts the silver rate.

Mutation testing automates this. StrykerJS (@stryker-mutator/core) injects small changes into the source, reruns the suite for each one, and reports a mutation score, which divides the mutants your tests caught (a test failed, or the run timed out) by the number of valid mutants. Its supported mutators flip comparison operators, negate booleans, swap arithmetic operators, empty string literals and remove block bodies. Runner plugins exist for Jest and for Vitest; check the plugin’s compatibility with your installed version before adopting it.

Fix Three: Read the Mocks Before the Assertions

Review a generated test diff in this order: what is faked, where the expected values came from, then what is asserted. What is faked determines what the test can observe. Where the expected value came from determines whether the assertion can disagree with the code. The assertion itself is the least informative line.

• A mock standing in for the unit under test: delete the test.
• An expected value that is a function call, or a literal with no rule behind it: recompute it from the requirement.
• Mocks only for real boundaries (database, network, clock) and assertions on the unit’s own output: keep it.

Keep the Judgement

AI is dependable at the mechanical parts of testing: fixtures, setup and teardown, parameterised tables of near-identical cases, boilerplate for async error paths. It is undependable at deciding which behaviours are worth asserting, because that decision lives in the requirement, not the code. Generate the scaffolding, then take the one test that matters most in the diff, break the line it should guard, and confirm it goes red before you merge.

FAQs

Does mutation testing replace code coverage?

No. The two report different gaps. Coverage tells you which lines never executed under any test; mutation testing tells you which executed lines no assertion protects. StrykerJS reports both: a mutant in unexecuted code gets the No coverage state, while a mutant in executed code that every test still passes is Survived. The mutation score counts both as undetected, so low coverage lowers the score directly.

Does StrykerJS work with Vitest and Jest?

Yes, through runner plugins. For Vitest, install @stryker-mutator/vitest-runner and set testRunner to vitest in the Stryker config; the plugin ships without Vitest itself, so read its package.json to find the oldest Vitest release it accepts. Tests that run through Vitest's Browser Mode fall outside what the runner handles. For Jest, use @stryker-mutator/jest-runner. Under the Vitest runner, Stryker switches Vitest's own coverage collection off and makes each mutant run stop at the first test that fails, because one failure is enough to kill the mutant.

How do I catch AI-generated tests that contain no assertion at all?

Make the framework fail any test with zero assertions. In Vitest, set expect.requireAssertions in the config or pass --expect.requireAssertions on the CLI; a test that finishes without calling expect fails. In Jest, call expect.hasAssertions() inside the test body, or from a beforeEach hook to apply it everywhere. Neither setting checks whether an assertion could fail, so a test that asserts a mock's own return value still passes.

Can AI-generated snapshot tests be trusted?

Not without review. A snapshot recorded from the current output asserts only that the code keeps producing whatever it produced when recorded, bugs included, which is the same tautology as a copied literal. Re-recording is one flag away: jest -u and vitest -u rewrite every failing snapshot. In CI, Jest's --ci flag fails on new snapshots instead of writing them silently. Review snapshot diffs against the requirement, not the previous output.