Skip to content

Handle SIGTERM gracefully in server and master - #262

Merged
beinan merged 3 commits into
lance-format:mainfrom
KaiqiJinWow:fix/sigterm-graceful-shutdown
Sep 23, 2026
Merged

beinan merged 3 commits into
lance-format:mainfrom
KaiqiJinWow:fix/sigterm-graceful-shutdown

Conversation

@KaiqiJinWow

Copy link
Copy Markdown
Contributor

Why

Kubernetes stops a pod by sending SIGTERM. The server and master previously waited only for Ctrl+C, so a rollout or scale-down could terminate either process before Axum drained active connections. For the server, this also meant state.shutdown() was never reached to close resident writers cleanly.

flowchart LR
    subgraph before ["Before"]
        direction TB
        ctrl_before["Ctrl+C / SIGINT"] --> graceful_before["Axum graceful shutdown"]
        term_before["SIGTERM"] --> exit_before["Process exits immediately"]
    end

    subgraph after ["After"]
        direction TB
        ctrl_after["Ctrl+C / SIGINT"] --> select_after["tokio::select!"]
        term_after["SIGTERM"] --> select_after
        select_after --> graceful_after["Axum graceful shutdown"]
        graceful_after --> drain_after["Drain active connections"]
        drain_after --> close_after["state.shutdown(): close writers"]
    end
Loading

For example, during a Kubernetes rolling deployment:

  1. Kubernetes sends SIGTERM to the old pod.
  2. The process now stops accepting new requests and lets in-flight requests finish.
  3. The server closes its writers before exiting with status 0.

What changed

  • Wait for either SIGINT/Ctrl+C or SIGTERM on Unix in both binaries.
  • Keep the existing Ctrl+C behavior on non-Unix platforms.
  • Log which signal initiated shutdown.
  • Verify that the Python server fixture exits successfully after Popen.terminate() sends SIGTERM on POSIX.

Testing

  • Server test suite: 80 passed.
  • Clippy for server and master, including all targets.
  • Rust formatting and Python lint checks.
  • Real-server SIGTERM smoke test: exited with status 0.

@beinan
beinan merged commit c62f0e7 into lance-format:main Sep 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants