Smoke Tests and Why Agents Keep Writing Them
Smoke tests explained: what to check, where to run them, what to exclude, and why coding agents keep adding them to CI and deploys.
A smoke test is a small set of checks that a deployed system is fundamentally alive: the process started, the main page returns 200, a user can log in, and the database answers a query. It does not judge whether the software is correct; it decides whether running the rest of the test suite is worth the time.
If you have shipped to CI for years without ever needing that definition, you are not alone. The phrase tends to arrive without an introduction, and lately it arrives inside a pull request: a smoke.sh or smoke.spec.ts that a coding agent added while finishing something else.
This article covers what belongs in a smoke test, where it runs, what a minimal one looks like in code, the filter for keeping things out of it, and why agents produce them so reliably.
Key Takeaways
- A smoke test checks that a deployed system is alive (boot, 200 on the main page, login, one real database read) and takes seconds, not minutes.
- It runs twice: as the first gate in CI, and immediately after a deploy against the environment that was actually deployed to; a localhost run cannot catch deployment failures.
- A check belongs in the smoke suite only if its failure blocks every user, every user passes through it, and it can break during deployment. Fewer than all three and it goes in regression.
- Coding agents write smoke tests because a finished task needs a cheap, binary, fast signal that nothing fundamental broke, which is exactly what a smoke test produces.
- Set a hard ceiling on the suite and move everything above it to regression; a twelve-minute smoke suite is a regression suite with the wrong name.
What Is a Smoke Test?
A smoke test answers one question, “is this build alive enough to test further?”, and its name is commonly traced to powering up new hardware and watching for smoke. In software the shape is the same: a fast, shallow pass across the paths every user depends on, with a binary outcome and no opinion about correctness.
It sits outside the usual testing pyramid rather than on a layer of it; the layers themselves are covered in Integration Tests vs End-to-End Tests and Unit vs Integration Testing in JavaScript: What to Use When. A smoke test is a gate in front of those suites, not a member of them.
Where Does Smoke Testing Run?
Smoke tests run in two places: as the first gate in CI before longer suites start, and immediately after a deployment, against the environment that was actually deployed to. The second placement is the one that earns its keep, and it is the one most often skipped.
Running a smoke test against localhost defeats its purpose, because the failures it exists to catch only happen at deployment: a missing environment variable, a migration that never ran, an asset bundle that never shipped. A process booted on the CI runner with APP_URL=localhost and a fresh in-memory database has none of those failure modes available to it. That kind of run is a boot check, and boot checks are useful, but they are a different and weaker thing. A local run against mocked services cannot fail for any of the reasons a deployment fails, so it should not carry the name.
The post-deploy run is also the natural trigger for rollback: AWS CodeDeploy runs your validation function once the new version is serving test traffic, and a failed result from it triggers a rollback. Keep the limit of that signal in mind, though. A green post-deploy smoke test proves the system answered, not that the user journey works; watching session replays of the first real sessions after a release is the technique that separates the two, because a checkout button that throws on a changed chunk hash never shows up in a status code.
What Does a Smoke Test Look Like?
A complete smoke suite can be one short script that hits the real deployed target with no mocks: a bounded wait for the health endpoint, one authenticated request, and one read that goes through the application to the database.
#!/usr/bin/env bash
# smoke.sh: runs against the deployed target in $APP_URL, never localhost
set -euo pipefail
: "${APP_URL:?set APP_URL to the deployed base URL}"
: "${SMOKE_USER:?}" "${SMOKE_PASS:?}"
status() { curl --silent --output /dev/null --write-out '%{http_code}' "$@"; }
# 1. Wait for the process to come up. This absorbs container start-up, nothing else.
for _ in $(seq 1 "${SMOKE_RETRIES:-10}"); do
[ "$(status "$APP_URL/health")" = "200" ] && break
sleep 3
done
[ "$(status "$APP_URL/health")" = "200" ] || { echo "health: not 200"; exit 1; }
# 2. Login. Assert the one status your app returns (200 for a JSON API, 302 for a form post).
code=$(status --data-urlencode "email=$SMOKE_USER" \
--data-urlencode "password=$SMOKE_PASS" "$APP_URL/login")
[ "$code" = "200" ] || { echo "login: got $code, expected 200"; exit 1; }
# 3. A real read through the app, with the credentials the app was deployed with.
body=$(curl --silent --fail "$APP_URL/api/products?limit=1") || { echo "query: request failed"; exit 1; }
[ -n "$body" ] || { echo "query: empty body"; exit 1; }
echo "smoke: ok"The status helper uses curl’s --write-out '%{http_code}' to capture the response code while --output /dev/null discards the body. set -euo pipefail makes the script exit on the first failure. The retry loop exists only to absorb start-up time after a deploy; it is not a way to paper over a flaky check.
Each assertion names one exact status. Every code in RFC 9110 carries a meaning, so a check that accepts “any response” is not a check: a 404 from a route that should exist is a deployment failure, and a 302 is only acceptable where the contract is a redirect. The third check matters more than it looks. Pinging the database server with a tool like pg_isready confirms the server accepts connections; a read through the application confirms the app can reach the database with the connection string, credentials, and schema it was deployed with, which is the failure class smoke tests exist for.
What Does Not Belong in a Smoke Test
Edge cases, business logic, anything slow, and anything flaky do not belong in a smoke test. A check belongs in the smoke suite only if all three conditions hold: its failure blocks every user from doing anything, every user passes through it, and it can break during a deployment. A check that meets fewer than three belongs in the regression suite. That three-question framing appears in at least one CI-testing vendor’s guidance, and it is a heuristic rather than a standard.
| Candidate check | Blocks every user? | Every user hits it? | Breaks on deploy? | Verdict |
|---|---|---|---|---|
| Health endpoint returns 200 | Yes | Yes | Yes | Smoke |
| Login with a test account | Yes | Yes | Yes | Smoke |
| Coupon code applies a discount | No | No | Yes | Regression |
| Admin CSV export downloads | No | No | Yes | Regression |
| Password reset email arrives | No | No | Yes | Regression |
Flakiness is disqualifying on its own. A smoke check that fails at random teaches the team to rerun red gates, and a gate people rerun until it passes is no longer a gate.
Why Do Coding Agents Keep Writing Smoke Tests?
Coding agents write smoke tests because an agent finishing a task needs a cheap, fast, unambiguous signal that it has not broken the whole system, and that is exactly the signal a smoke test produces. An agent that just edited code cannot afford the full suite on every iteration and cannot judge correctness by inspection, so it reaches for the check that answers “is it still alive” in seconds and returns a clean exit code.
This is not accidental. Anthropic’s Claude Code documentation tells developers to hand the agent something it can run to check its own work, whether that is a test suite, a build, a linter, or a small script. Given a signal it can read for itself, the agent keeps working and re-checking without waiting for a person to spot the mistake. A smoke script fits that description closely, which is why agent-authored repositories tend to grow one even when the team never used the term. That is the reason developers are meeting “smoke test” now, often by finding one in a diff they did not write.
When one appears in a PR, review it against four questions. Does it read the target URL from an environment variable rather than hardcoding localhost? Does it assert a specific status instead of “not an error”? Does it finish in seconds? Does every check pass the three-question filter above? An agent-written smoke test that fails any of these is either a boot check or a regression test wearing the wrong label.
The Failure Mode: The Suite Grows
A smoke suite stops being a gate the moment it grows past a few seconds. The pattern is predictable: every feature adds a check “just in case,” an agent adds another after every task, and one day the gate takes twelve minutes and people start skipping it. At that point it is a slow regression suite under the wrong name.
The fix is a hard ceiling with everything above it moved to regression. Our recommended default is a handful of checks, on the order of five, that finish in seconds; TestingXperts puts the outer limit at ten minutes, past which teams start skipping the gate. The exact number matters less than having one written down and enforced in review, including review of agent-authored additions.
Conclusion
A smoke test is the smallest possible proof that a deployment is alive: health, login, one real read, run against the environment you actually shipped to, finished in seconds. Everything else is regression. The next time an agent hands you a smoke.sh, check that it targets a real URL, asserts exact statuses, and stays under the ceiling, then wire it to run after every deploy.
FAQs
What is the difference between a smoke test and a health check?
A health check is one endpoint that an orchestrator or load balancer polls to decide whether to route traffic or restart a container; Kubernetes calls these liveness and readiness probes. A smoke test runs once per deployment, calls that endpoint plus a login and a database read, and returns an exit code that gates the pipeline. The health endpoint is the smoke test's first assertion, not a replacement for it.
What is the difference between smoke testing and sanity testing?
Smoke testing is broad and shallow: it checks that the core paths of a build are alive before deeper testing starts. Sanity testing, in conventional QA terminology, is narrow and deep: it verifies a specific fix or change works on a build that has passed smoke, and often counts as a subset of regression testing. In a CI pipeline the smoke suite is the gate; sanity checks belong with regression.
Should smoke tests run against production, and is that safe?
Yes. The post-deploy run should target the environment users actually reach, including production, because deployment failures only appear there. Keep it safe by using a dedicated seeded test account supplied through CI secrets, limiting checks to a login and read-only requests, and excluding anything that writes data or sends email. If a write is unavoidable, scope it to a test tenant and clean it up in the same script.
Can I write a smoke test in Playwright or Cypress instead of a shell script?
Yes, provided it follows the same rules: read the base URL from an environment variable, assert exact statuses or visible elements, and finish in seconds. Browser runners add start-up and page-load time per check, so keep the browser smoke suite to one or two journeys and leave HTTP checks in curl. Put the smoke spec in its own file so CI can run it without the rest of the suite.
