Kubernetes Health Checks

Benzene.HealthChecks provides purpose-built liveness/readiness convenience methods on top of the general-purpose health checks support, matching Kubernetes' own liveness/readiness/ startup probe model. This page covers the semantics, how to wire them up per transport, and example probe configuration. Read Health Checks first if you haven't already — this page assumes you know IHealthCheck/IHealthCheckBuilder and doesn't repeat that material.

Liveness vs. readiness

Kubernetes' guidance is specific about what belongs in each, and Benzene doesn't second-guess it:

Liveness answers "is this process itself still working?" A liveness probe failure causes Kubernetes to restart the pod. Keep liveness checks cheap and local — verify the process can still do work at all (e.g. an internal loop isn't stuck), not that every external dependency is reachable. Checking a database or downstream service in a liveness probe is a well-known anti-pattern: a flaky dependency would cause Kubernetes to restart pods that are actually fine, potentially triggering a restart storm across your whole fleet in response to one struggling downstream service.
Readiness answers "is this instance ready to receive traffic right now?" A readiness probe failure only removes the pod from the Service's endpoint list — no restart, traffic just stops routing to it until it passes again. Readiness is for instance-local conditions: warmup not finished, this pod's connection pool is rebuilding, this pod is shedding load, this pod is draining for shutdown (ShutdownReadinessHealthCheck).

Caution

Don't reflexively put every external-dependency check in readiness. A check against a shared downstream (a queue, a database, an API that every replica talks to) is shared-fate: all replicas run the same check against the same dependency, so a transient blip fails all their readiness probes at once and Kubernetes pulls every pod from the Service. The Service then has zero endpoints — callers get connection-refused / DNS failures instead of a structured 503 with Retry-After, which breaks L7 retries and circuit breakers and turns a degradation into a total outage. De-routing only helps when some replicas are healthy to shed to; for a shared dependency there are none. This is the classic cascading-failure anti-pattern. Gate readiness on a dependency only when you've reasoned that it's a hard, synchronous dependency you truly cannot serve any traffic without, and that failing fast at the load balancer is better than returning a 503 — e.g. a single-region synchronous API. When in doubt, use the deep healthcheck layer (below) instead.

For a dependency you have reasoned is safe to gate on, register it explicitly under readiness: Benzene.HealthChecks.EntityFramework's DatabaseConnectionHealthCheck / Benzene.HealthChecks.Http's HttpPingHealthCheck etc. via .UseReadinessCheck(...).

Auto-wired client checks land on the deep `healthcheck` layer, not a probe

When a Benzene client is configured (e.g. .UseSqs(queueUrl)) it auto-registers a non-destructive reachability check for that dependency — but on the general healthcheck topic only, never a liveness or readiness probe. That deep layer is scraped by monitoring / the mesh / humans and triggers no automated Kubernetes action, so "every client ships a check" gives you visibility without the shared-fate cascading-failure risk above. Opt out per client with healthCheck: false (e.g. .UseSqs(queueUrl, healthCheck: false)), or promote a specific dependency to readiness yourself if you've reasoned it safe. Point a Kubernetes probe at livez/readyz, and your monitoring/mesh at the healthcheck endpoint — not the other way around.

Client / contract-drift checks belong in neither probe

The downstream service check (AddServiceCheck — the consumer side of the runtime contract-drift check, which sends benzene:healthcheck to a provider and optionally compares its contract hash against a generated client's HashCode) is not an ordinary external-dependency check, and it does not belong in a liveness or a readiness probe. It calls a downstream provider's health endpoint and compares contract hashes, so it fails the two tests above harder than a database check does:

It is transitive. The thing it checks — another service's health endpoint — may itself aggregate that service's dependencies and clients. Put it in a liveness probe and one slow leaf service restarts healthy consumer pods across the fleet (a restart storm one hop removed from the actual problem); put it in a readiness probe and the outage propagates upstream — the failing provider's consumers look unready to their consumers, de-routing an entire dependency chain over a single leaf failure.
Contract drift is a versioning signal, not a serve-traffic signal. A drifted-but-working provider reports Warning, which deliberately does not flip IsHealthy. A pod that is one contract revision behind can still serve traffic perfectly — restarting it or pulling it from rotation over that annotation is never the right response.

So keep the downstream check off /livez and /readyz. Benzene.HealthChecks provides a dedicated contracts diagnostic topic for exactly this — a probe-less surface Kubernetes never points at, which the mesh / your alerting consume instead. Register your downstream checks with UseContractsCheck + AddServiceCheck (from Benzene.Clients.HealthChecks):

using Benzene.HealthChecks;
using Benzene.Clients.HealthChecks;

app.UseContractsCheck(x => x
    // Library-provided: sends benzene:healthcheck over IBenzeneMessageSender (resolved from the
    // container), so nothing is generated or hand-written for it. The hash is optional - supply a
    // generated client's HashCode for drift reporting, omit it for reachability only.
    .AddServiceCheck("OrderService", new OrderServiceClient(sender).HashCode)
    .AddServiceCheck("PaymentService"));

Because the check sends benzene:healthcheck, the consumer must register an outbound route for that topic aimed at the provider — an explicit opt-in per dependency, over a transport that can actually answer. (AddContractCheck<TClient>(...) remains for a hand-written IHasHealthCheck client, for the unusual case where the standard call isn't what you want.)

UseContractsCheck answers only the contracts topic (Constants.DefaultContractsTopic) — like UseLivenessCheck/UseReadinessCheck it does not also match the generic healthcheck topic, and no probe is pointed at it. Each ClientHealthCheck reports the downstream contract relationship: reachable + matching contract is Ok, reachable + drifted is Warning (degraded-but-not-fatal, does not flip IsHealthy), and only an unreachable provider is Failed — so contract drift surfaces as a mesh drift badge (see Mesh UI) or an alert, never a restart. The one narrow exception is a hard synchronous dependency you genuinely cannot serve any traffic without — a targeted reachability-only check against that one provider may go in readiness (never liveness); that is AddServiceCheck("ThatProvider") with no hash, so the contract-drift portion is excluded, drift never being a reason to stop serving traffic. See work/archive/client-health-checks-design-2026-08.md for the full rationale.

Topic-based wiring (every transport): `UseLivenessCheck` / `UseReadinessCheck`

using Benzene.HealthChecks;
using Benzene.HealthChecks.EntityFramework;

app.UseLivenessCheck(x => x
    .AddHealthCheck<ProcessResponsiveCheck>()); // your own check, cheap, no external I/O

app.UseReadinessCheck(x => x
    .AddHealthCheckFactory(new DatabaseHealthCheckFactory<MyDbContext>("20250101000000_Latest"))
    .AddHttpPing("https://downstream-service/health"));

Each responds only to its own topic (Constants.DefaultLivenessTopic = "liveness", Constants.DefaultReadinessTopic = "readiness") — unlike UseHealthCheck(), neither also matches Constants.DefaultHealthCheckTopic. This is deliberate: if both did, whichever was registered first in the pipeline would silently swallow every request for the shared "healthcheck" topic, and the other would never run for it.

HTTP-path wiring: `Benzene.Aws.Lambda.ApiGateway`

This package deals with raw HTTP requests directly, so it exposes method+path matching instead of (in addition to) topic matching — defaulting to the conventional Kubernetes probe paths:

app.UseLivenessCheck(x => x.AddHealthCheck<ProcessResponsiveCheck>());              // GET /livez
app.UseReadinessCheck(x => x.AddHealthCheckFactory(dbHealthCheckFactory));           // GET /readyz

// override the path if your ingress/probe config expects something else
app.UseLivenessCheck("/healthz/live", x => x.AddHealthCheck<ProcessResponsiveCheck>());

ASP.NET Core

ASP.NET Core doesn't have an equivalent HTTP-path overload (an ASP.NET Core request's topic is already resolved via routing before Benzene's middleware runs — see Health Checks for why) — use the topic-based UseLivenessCheck/UseReadinessCheck there, and register an IHttpEndpointDefinition mapping the conventional paths to the liveness/readiness topics so an incoming GET /livez/GET /readyz actually resolves to the right topic:

using Benzene.HealthChecks;
using Benzene.Http.Routing;

public class MyStartUp : BenzeneStartUp
{
    public override void ConfigureServices(IServiceCollection services, IConfiguration configuration)
        => services.UsingBenzene(x => x
            .AddMessageHandlers()
            .AddSingleton<IHttpEndpointDefinition>(_ =>
                new HttpEndpointDefinition("GET", "/livez", Constants.DefaultLivenessTopic))
            .AddSingleton<IHttpEndpointDefinition>(_ =>
                new HttpEndpointDefinition("GET", "/readyz", Constants.DefaultReadinessTopic)));

    public override void Configure(IBenzeneApplicationBuilder app, IConfiguration configuration) => app
        .UseHttp(http => http
            .UseLivenessCheck(x => x.AddHealthCheck<ProcessResponsiveCheck>())
            .UseReadinessCheck(x => x.AddHealthCheckFactory(new DatabaseHealthCheckFactory<MyDbContext>("...")))
            .UseMessageHandlers()); // your regular handlers, if any
}

No IMessageHandlerDefinition registration is needed for the health check routes — UseLivenessCheck/ UseReadinessCheck are raw middleware that intercept the request directly by topic, before .UseMessageHandlers()'s own handler-resolution would otherwise run.

Example Kubernetes manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-benzene-service
spec:
  template:
    spec:
      containers:
        - name: my-benzene-service
          image: my-registry/my-benzene-service:latest
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

Both probes work correctly against Benzene's response because the HTTP status code — not just the JSON body — reflects health: 200 when isHealthy is true, 503 Service Unavailable when it is false, with the health report as the body either way (see HTTP status codes). Kubernetes' httpGet probe type only inspects the status code (2xx-399 is a pass, anything else is a failure), so this matters — a health check that returned 200 unconditionally, with the real status only visible in the response body, would never actually fail a Kubernetes probe.

If you'd rather probe over gRPC, see Health Checks — gRPC (grpc.health.v1) for Benzene.Grpc.AspNet's standard-protocol bridge, which Kubernetes' native gRPC probe type (grpc.livenessProbe/readinessProbe.grpc) can query directly.

gRPC liveness/readiness split

By default Benzene.Grpc.AspNet's bridge aggregates every registered IHealthCheck into the single overall grpc.health.v1 service (empty service name). That's fine for a readiness probe or a plain "is it up" check — but a liveness probe pointed at it would run your external-dependency (readiness) checks too, exactly the restart-on-flaky-dependency anti-pattern the liveness/readiness split exists to avoid.

To get the same split over gRPC that HTTP UseLivenessCheck/UseReadinessCheck give you, set LivenessCheckTypes/ReadinessCheckTypes on BenzeneGrpcOptions — lists of the check Types that belong in each. Benzene then publishes named grpc.health.v1 services "liveness" and "readiness" alongside the overall "" service, each reporting only its own subset:

services.AddBenzeneGrpc(o =>
{
    o.EnableHealthChecks = true;
    o.LivenessCheckTypes = new[] { "ProcessResponsive" };          // cheap/local checks only
    o.ReadinessCheckTypes = new[] { "Database", "HttpPing" };       // external-dependency checks
});

Kubernetes' native gRPC probe type can then target the right service name:

livenessProbe:
  grpc:
    port: 8080
    service: liveness
readinessProbe:
  grpc:
    port: 8080
    service: readiness

When neither is set, behaviour is unchanged: a single aggregate "benzene" check on the overall "" service.

What this does not cover

Startup probes. Kubernetes' third probe type (for slow-starting containers) has no dedicated Benzene wiring — reuse UseReadinessCheck (or a separate topic/path of your own) for it; there's nothing startup-probe-specific in Benzene today.
Graceful shutdown coordination. Flipping readiness to unhealthy during pod termination (so Kubernetes stops routing new traffic while in-flight requests drain) is available as a building block: ShutdownReadinessHealthCheck + ShutdownState.LinkTo(CancellationToken) — wire the latch to your host's shutdown token (IHostApplicationLifetime.ApplicationStopping, which fires on SIGTERM) and add the check to your readiness probe. See ShutdownReadinessHealthCheck. Benzene still doesn't auto-wire it to the host lifecycle for you (that one LinkTo line is yours), and pair it with a preStop sleep / terminationGracePeriodSeconds so kube-proxy has removed the endpoint before the process exits.