12k
All articles

Shutting Down a Node Server Gracefully

Graceful Node shutdown for SIGTERM: handle readiness, drain requests with server.close, close resources in order, and avoid rollout 502s.

OpenReplay Team
OpenReplay Team
Shutting Down a Node Server Gracefully

A graceful Node shutdown handles SIGTERM in a fixed order: fail the readiness check, wait for the load balancer to stop routing traffic, drain in-flight requests with server.close(), close resources in dependency order, then exit.

If you already do most of that and still see 502s and connection resets on every rollout, the handler itself is often not where the problem is. Three things around it tend to cause it: the signal never reaches Node, the drain never finishes, or the orchestrator kills the pod before the drain is done. This article builds one small handler, works through all three, and ends with a test you can run.

Key Takeaways

  • Without a SIGTERM handler, Node exits immediately and every in-flight request ends as a connection reset or truncated response.
  • Await server.close() before closing the database pool; closing the pool first turns every in-flight query into an error.
  • Since Node 19.0.0, server.close() closes idle keep-alive connections itself; the old “close callback never fires” hang is historical.
  • A shell-form Dockerfile CMD means the shell, not Node, receives SIGTERM, so no handler code ever runs.
  • Set an in-process forced-exit timer below terminationGracePeriodSeconds so the process chooses its exit instead of receiving SIGKILL.

What Happens Without a SIGTERM Handler?

Node’s default response to SIGTERM is to terminate the process, per the signal events documentation. Whatever was in flight at that instant is cut off mid-response: the client sees a reset or a partial body, and the load balancer logs a 502. In session replay this failure has a recognisable shape: a user action that spins and then errors, clustered in a narrow window that matches a rollout rather than anything the user did. That client-side view is usually where the bug is first noticed.

The Correct Node Graceful Shutdown Order

A correct handler does five things in order: flip a shutdown flag so readiness fails, wait briefly for the load balancer to react, await server.close() so in-flight requests finish, close remaining resources, then exit. The await on that close is the part that matters. A handler that calls server.close() without waiting for its callback, cleans up resources, and exits has re-created the crash it was written to prevent.

  1. Flip a shutdown flag so the readiness probe returns 503.
  2. Wait briefly for the load balancer to stop routing new traffic.
  3. Await server.close() so in-flight requests finish.
  4. Close the remaining resources in dependency order.
  5. Exit the process.
// server.js (Express)
const { setTimeout: sleep } = require('node:timers/promises');

let shuttingDown = false;

app.get('/health/live', (req, res) => res.sendStatus(200));
app.get('/health/ready', (req, res) => res.sendStatus(shuttingDown ? 503 : 200));

async function shutdown() {
  if (shuttingDown) return;
  shuttingDown = true;                    // readiness now returns 503

  await sleep(LB_WAIT_MS);                // matches your preStop sleep

  await new Promise((resolve, reject) =>
    server.close((err) => (err ? reject(err) : resolve()))
  );

  await closeResources();                 // next section
  process.exit(0);
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

Why Can the Drain Stall?

On any supported Node release, server.close() stops new connections, closes idle keep-alive sockets, and waits for connections with a request in flight to finish. The widely repeated warning that idle keep-alive sockets keep the close callback from ever firing describes Node before 19.0.0, so the hang no longer occurs on supported versions. If you must run older runtimes, call server.closeIdleConnections() (added in Node 18.2.0) immediately after initiating the close, not before it, to avoid racing newly created connections.

What still stalls a drain is legitimately active work: slow handlers, server-sent events, and sockets upgraded to another protocol. That is what the forced-exit timer below exists for.

Close Resources in Dependency Order

Close the HTTP server first, then queue workers and background jobs, then Redis, then the database pool. The order follows the dependency chain: request handlers and jobs use Redis and the pool, so closing the pool while they are still running turns every in-flight query into an error, which defeats the drain you just awaited.

async function closeResources() {
  await worker.close();   // BullMQ or similar: finish the active job
  await redis.quit();
  await pool.end();       // pg pool last: nothing queries after this
}

Jobs that run longer than the grace period need checkpointing so they can resume, not draining.

Which Container Settings Break Graceful Shutdown?

Node must be the process that actually receives the signal, or nothing above matters. The Dockerfile reference is explicit that the shell form runs your command under /bin/sh -c, which does not pass signals to its child, so a shell-form CMD means the SIGTERM from docker stop never reaches Node.

# Broken: /bin/sh -c receives SIGTERM, node never does
CMD node server.js

# Correct: node is the signal target
CMD ["node", "server.js"]

If you need an init process to reap zombies, tini forwards signals to its child, so the handler still runs. Note that docker stop escalates to SIGKILL after 10 seconds by default on Linux, configurable with -t.

In Kubernetes, set terminationGracePeriodSeconds (default 30) above your total drain time, and add a preStop sleep: endpoint removal is evaluated in parallel with SIGTERM and propagates asynchronously via EndpointSlices, so requests keep arriving for a moment after the signal. Keep the liveness probe passing while readiness fails; a liveness probe wired to the same failing endpoint gets the container restarted mid-drain.

terminationGracePeriodSeconds: 30   # > preStop + LB wait + drain + cleanup
lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]       # LB_WAIT_MS should match

Choose Your Exit Before SIGKILL Does

Arm a forced-exit timer at the top of the handler, set below the orchestrator’s grace period, so a stalled drain ends with your log line and exit code instead of SIGKILL. As the last resort it calls server.closeAllConnections() (added in Node 18.2.0), which tears down every open connection, including ones still handling a request. Connections that have switched to another protocol survive it, so WebSockets need their own close-frame broadcast.

const forceExit = setTimeout(() => {
  server.closeAllConnections();
  process.exit(1);
}, GRACE_MS - 2_000);   // grace period minus a buffer, never a fixed default
forceExit.unref();

How Do You Prove the Shutdown Works?

The acceptance test is mechanical: spawn the server, fire a request at a deliberately slow route, send SIGTERM mid-flight, and assert the response still arrives with 200 and the process exits 0.

// verify-shutdown.js
const { spawn } = require('node:child_process');
const assert = require('node:assert');

const child = spawn('node', ['server.js'], { stdio: ['ignore', 'pipe', 'inherit'] });
child.stdout.on('data', async (chunk) => {
  if (!chunk.toString().includes('listening')) return;
  const pending = fetch('http://localhost:3000/slow'); // route awaits ~2s
  setTimeout(() => child.kill('SIGTERM'), 100);
  const res = await pending;
  assert.equal(res.status, 200);
  const code = await new Promise((r) => child.on('exit', r));
  assert.equal(code, 0);
  console.log('graceful shutdown verified');
});

If this passes locally but rollouts still drop requests, the remaining suspects are the container layer: shell-form CMD or a grace period smaller than your drain budget.

The handler itself is forty lines; the reliability comes from the ordering and the environment around it. Wire the test into CI so the next refactor cannot silently reintroduce the reset-on-deploy bug you just fixed.

FAQs

Can a Node.js process catch or handle SIGKILL?

No. Node refuses to attach a listener for SIGKILL, and the signal ends the process on every platform whatever your code says, so no cleanup runs. SIGSTOP cannot be listened for either. This is why the in-process forced-exit timer matters: it must fire before the orchestrator's grace period expires, so the process drains and exits on SIGTERM instead of being killed with no chance to respond.

What is the difference between closeIdleConnections and closeAllConnections?

Both methods were added in Node 18.2.0. closeIdleConnections only shuts sockets that are sitting idle between requests, so anything mid-request is left to finish. closeAllConnections is the blunt one: it tears down every open connection, including ones still handling a request, though connections that have switched to another protocol such as WebSocket survive it. Since Node 19.0.0, server.close clears idle connections on its own, so closeIdleConnections is only worth calling if you still support older runtimes.

Does process.exit wait for in-flight requests or pending async work?

No. process.exit shuts the process down at once and drops whatever async work is still queued, right down to output that has not finished being written to stdout or stderr. That is why the shutdown handler awaits server.close and resource cleanup before calling exit; calling exit any earlier recreates the dropped-request crash the handler exists to prevent. Outside a handler, prefer setting process.exitCode and letting the process exit naturally.

Do SIGTERM handlers work on Windows?

Not the same way. Windows has no POSIX signals, and the Node.js docs list SIGTERM as unsupported there, even though your code can still register a listener for it. Ctrl+C does raise SIGINT everywhere, which is why the handler also listens for SIGINT for local development. Test the SIGTERM drain path inside a Linux container, where docker stop and Kubernetes actually deliver the signal.

Understand every bug

Uncover frustrations, understand bugs and fix slowdowns like never before with OpenReplay — self-hosted, with full data ownership.

Star on GitHub

We use cookies to improve your experience. By using our site, you accept cookies.