Grace

The library for exiting gracefully.

Problem

When running in Kubernetes we have to consider certain timings in terms of how long it takes for the whole networking system to realise that an application is not, or should no longer be available. On top of that, Kubernetes can just send any application a shutdown command, which we must respect, but still complete our work. This library should address both of these issues.

The shutdown flow

sequenceDiagram
    participant U as User / OS
    participant P as Pod / App
    participant K as K8s / kubelet
    participant E as Envoy / LB

    K->>P: /ready?
    P-->>K: /ready -> true
    U->>P: SIGTERM / SIGINT
    K->>P: /ready?
    P-->>K: /ready -> false

    K->>P: /live?
    P-->>K: /live -> true

    critical Request Accept Grace Period
        K->>E: Remove the endpoint
        E-->>K: Endpoint removed
    end

    P->>P: Srv.Stop() called
    K->>P: /live?
    P-->>K: /live -> true

    critical Shutdown Time
        P->>P: Shutdown work
    end

    K->>P: SIGKILL / pod terminated

The /ready endpoints explain whether the service is ready to accept traffic, should only return true if the service AND its dependencies are ready. The /live endpoint simply indicates whether the service is alive, and should return true as long as there is an HTTP server running, even if the service is not ready to accept traffic.

Once a SIGTERM/SIGINT is received the /ready endpoint should return false, but the /live should continue to return true. Then for the length of the Request Accept Grace Period(RA GP) the service should continue to accept and process requests. After RA GP has passed, the HTTP server should shutdown, /live should return false and the service should complete any shutdown procedures. It has Shutdown Time(ST) to do so. After which Kubernetes will forcefully kill the process.

In Kubernetes this entire time frame is represented as a single timeframe called terminationGracePeriodSeconds. This should be set to RA GP + ST.

Installation

go get github.com/corticph/gocomo/grace

Usage

grace exposes a single entrypoint: the resource manager. Register your dependencies, expose the probe server, then block on Start. On SIGINT/SIGTERM it flips /ready to unavailable, keeps serving for the request-accept grace period, shuts every resource down, and exits.

A dependency only needs to satisfy Resource (Name, Ping, Shutdown):

type DB struct{ conn *sql.DB }

func (db *DB) Name() string                       { return "postgres" }
func (db *DB) Ping(ctx context.Context) error     { return db.conn.PingContext(ctx) }
func (db *DB) Shutdown(ctx context.Context) error { return db.conn.Close() }

Wire it up in main:

func main() {
    ctx := context.Background()

	logger := slog.Default()

	rm, err := grace.NewResourceManager(grace.WithLogger(logger))
	if err != nil {
		logger.Error("failed to create resource manager", "error", err)
		os.Exit(1)
	}

	rm.AddResource(db)       // probed by /ready and shut down on exit
	rm.AddProbe(cache)       // readiness only
	rm.AddShutdowner(worker) // shutdown only

	// Serve /ready and /live first so Kubernetes can reach them immediately.
	probes := rm.NewHttpServer()
	go func() {
		if err := probes.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			logger.Error("probe server failed", "error", err)
		}
	}()

	// ... start your application server in a go func ...

    rm.AddResource(appServer)

	// Blocks until a shutdown signal, then runs the graceful shutdown flow and exits.
	rm.Start(ctx)
}

Configuration

Every timeout and endpoint has a sane default and is overridable via environment variables:

Set Kubernetes' terminationGracePeriodSeconds to REQUEST_ACCEPT_GRACE_PERIOD + SHUTDOWN_GRACE_PERIOD.