Diagnosing Failures

A message failed in production. How do you find out why?

This page ties together the signals Benzene emits when something goes wrong in a message handler or in the middleware pipeline: what you get out of the box with nothing configured, the recommended middleware stack that makes failures easy to trace, the log fields each layer adds, and how the same failure shows up across logs, traces, and metrics. For setting up observability backends see Monitoring & Diagnostics; for catching exceptions and mapping them to responses see Global Error Handling.

Wiring problems are caught before any message arrives

Most of what follows is about diagnosing a message that failed. A whole class of problem is not that at all: the wiring is wrong, and every message will fail the same way for the same reason. Those are checked once, at start-up, before the first message.

Every host runs the checks — AWS Lambda, ASP.NET Core, the generic host, Azure Functions — and so does BenzeneTestHost, which means a wiring mistake is a red unit test rather than something you meet on a deployed function.

When terminal-middleware is wrong

The rule is the ITerminalMiddleware marker: a pipeline needs at least one middleware that says it can end one. "Does this middleware call next?" is not answerable without running it, and most middleware answers "sometimes", so the marker is a statement of intent rather than an inference.

Everything in Benzene that ends a pipeline carries it — UseMessageHandlers(), every transport router, every client's send middleware, the health-check endpoints, the spec and mesh UIs. Your own middleware does not, so a pipeline that ends in yours is the one false positive this check can produce. Say so and it stops:

public class ServeFromCache : IMiddleware<SqsMessageContext>, ITerminalMiddleware { ... }

Nothing about execution changes — the marker is read once, at start-up. For inline middleware, where there is no type to mark, UseTerminal(name, ...) is the same statement:

app.UseTerminal("static-response", async (context, next) => { ... });

If a check is wrong for your application, one switch covers all of them:

services.UsingBenzene(x => x
    .AddBenzene()
    .AddBenzeneStartUpChecks(BenzeneStartUpCheckMode.Advisory)   // log and continue
    .AddMessageHandlers(typeof(Program).Assembly));

BenzeneStartUpCheckMode.Disabled turns them off entirely. There is deliberately one switch rather than one per check: an escape hatch you have to go looking for is not an escape hatch.

Two related checks are worth knowing about but are not part of this phase:

• Container validation. new MicrosoftServiceResolverFactory(services, validateOnBuild: true) turns on ValidateOnBuild and ValidateScopes, which catch an unconstructible registration (a missing .AddBenzene(), a handler dependency you forgot) at container-build time with a better message than the runtime path. It is opt-in because a partially-composed container is a supported arrangement here — one codebase often builds several deployables that each mount a subset — and a check that rejects a valid one is worse than the bug it catches. Turn it on for a fully-composed application.
• Compile-time. BENZ001 (duplicate [Message] topic) and BENZ002 ([HttpEndpoint] with no [Message]) are compiler errors, delivered with Benzene.Core.MessageHandlers. They catch the attribute-declared half of the same two mistakes before you run anything at all.

Infrastructure failures are not the message's fault

There is a third kind of failure that is neither of the two below, and it needs the opposite handling: the service is mis-wired. A registration is missing, so every message fails the same way, and no amount of retrying or redelivery will change that.

Reported per message, that is a slow disaster. Each record is dead-lettered on its own, the next one does exactly the same, and the function keeps returning success — so the queue drains into the DLQ while the service reports itself healthy.

Benzene now tells the two apart. A resolution failure throws BenzeneResolutionException, and BenzeneFailure.IsInfrastructure(exception) recognises it anywhere in the exception chain:

• SQS fails the whole invocation. The exception escapes per-record handling entirely, so SQS retries the batch instead of dead-lettering it a message at a time, and the function fails loudly and repeatedly until the wiring is fixed. This overrides SqsBatchFailureMode — but only for infrastructure failures. A handler that throws on one payload keeps partial-batch reporting exactly as before.
• Every transport logs it distinctly, prefixed [benzene:infrastructure], so a wiring fault is greppable and separable from the per-message noise around it.

The classification is deliberately narrow: only a proven resolution failure counts, because the cost of a false positive is failing a whole batch over one bad message. Anything unrecognised is treated as a business failure — the existing, safe behaviour.

Most wiring faults never get this far, because the pipeline-resolution start-up check above constructs every middleware before any message arrives. What reaches here is the residual: a handler's own dependency, which is only constructed when a message is dispatched to its topic.

Two kinds of failure

Benzene distinguishes between two failure modes, and they surface differently:

• An unsuccessful result — a handler returns BenzeneResult.NotFound(...), BenzeneResult.BadRequest(...), BenzeneResult.UnexpectedError(...), etc. This is a normal return value: the response pipeline serializes an ErrorPayload and maps the status to the transport (an HTTP 404/400/500, an SQS batch-item-failure, ...). Nothing threw.
• A thrown exception — a handler (or middleware, or a mapper) throws. This bypasses the response pipeline entirely; how it settles depends on the transport (see the catch matrix).

The distinction matters when you're reading logs: an unsuccessful result and an exception are different events with different log lines, and a stack trace only exists for the second kind.

What you get with nothing wired

Even with no logging or diagnostics middleware added, Benzene emits a baseline error signal so failures aren't completely silent:

These are enough to answer "did my handler run, and did it fail?" from a plain log tail — but they carry no correlation id, no trace id, and (for the baseline) no per-message processing time. Wiring the recommended stack below gives you all of that.

These log lines only reach a sink if a logging provider is configured. UsingBenzene(...) calls AddLogging() so ILogger<T> resolves, but with no provider every log call is a no-op. Add a provider (AddLogging(x => x.AddConsole()), Serilog, Application Insights, ...) or you'll see nothing regardless of what middleware you add. See Monitoring — Logging.

The recommended stack

Add these to each transport pipeline, in this order, before .UseMessageHandlers():

app.UseSqs(sqsApp => sqsApp
    .UseW3CTraceContext()      // 1. FIRST — continue the caller's distributed trace
    .UseBenzeneEnrichment()    // 2. attach invocationId/traceId/topic/transport/handler to every log
    .UseExceptionHandler((SqsMessageContext ctx, Exception ex) => ctx.IsSuccessful = false) // 3. catch + log thrown exceptions
    .UseLogResult(_ => { })    // 4. one structured log line per message (Info on success, Error on throw)
    .UseMessageHandlers());

and, once, in ConfigureServices:

services.UsingBenzene(x => x
    .AddMessageHandlers(typeof(HelloWorldMessageHandler).Assembly)
    .AddDiagnostics());        // Activity span per middleware — marks failing spans with Error status

What each layer contributes when something fails:

Add UseBenzeneMetrics() too if you want the failure to also move a counter/histogram — see below.

How a failure shows up across the three signals

The same failing message produces a coordinated picture once the stack is wired:

In logs

A thrown InvalidOperationException("boom") handling order:create on SQS, with the stack above:

[Error] Benzene: BenzeneResult faulted
    System.InvalidOperationException: boom
    { processTime=42, invocationId=..., traceId=4bf92f..., topic=order:create, transport=sqs, handler=CreateOrderHandler }

[Error] Benzene: Unhandled exception caught in middleware pipeline
    System.InvalidOperationException: boom
       at MyApp.CreateOrderHandler.HandleAsync(...)
    { invocationId=..., traceId=4bf92f..., topic=order:create, transport=sqs, handler=CreateOrderHandler }

UseLogResult sits inside UseExceptionHandler, so the innermost catch logs first ("faulted", with processTime), then the exception handler's net logs and settles the message. Both lines describe the same throw from different layers — the traceId ties them together.

An unsuccessful result (no throw) instead produces a single Warning from the router:

[Warning] Benzene: Handler CreateOrderHandler for topic order:create returned unsuccessful status not-found
    { invocationId=..., topic=order:create, transport=sqs, handler=CreateOrderHandler }

Grep tips: traceId ties every line of one message together (and across services); topic + status narrows to a failure kind; transport separates which adapter handled it.

In traces

With AddDiagnostics() and an OTel exporter (Monitoring — OpenTelemetry), the failing middleware's span carries status = Error and an exception event with the stack, so a trace viewer (Jaeger, Tempo, Application Insights) points straight at the stage that threw — the MessageRouter span for a handler throw. Every span is tagged with the topic/handler, and UseW3CTraceContext() keeps it on the caller's trace.

In metrics

With UseBenzeneMetrics(), every message moves two instruments on the "Benzene" meter, tagged topic/transport/result:

• benzene.messages.processed (counter) — the result tag lets you alert on a rising rate of a specific failure status for a topic
• benzene.message.duration (histogram, ms) — a failure that's actually a timeout shows up here as a latency spike, not just an error count

Ordering footguns

A few middleware silently do nothing if wired in the wrong order — no error, just missing data:

• UseW3CTraceContext() must be the first middleware. Everything after it inherits the correct ambient Activity.Current parent; put it later and the spans before it start a new, disconnected trace and inbound traceparent continuation is lost.
• Enrichment's invocationId needs UseBenzeneInvocation() upstream. On the batch/per-message transports (SQS, SNS, Kafka, Event Hub) this is wired automatically per record; on a hand-built pipeline, if you want invocationId you need the invocation populated before UseBenzeneEnrichment() runs, or that one field is simply omitted.
• UseExceptionHandler(...) only wraps what comes after it. Register it before .UseMessageHandlers() (and before anything else whose throws you want caught). Anything earlier in the chain is unprotected.
• UseLogResult(...) should sit outside the handlers so its processTime covers the whole pipeline and it logs the result the handlers produced.

For the first of these, there's an opt-in startup check: IServiceResolver.LogPipelineOrderingIssues(builder) (Benzene.Diagnostics) warns if UseW3CTraceContext() isn't the first middleware in a pipeline. Call it once after building a pipeline; it's advisory and never throws. (The enrichment/invocation rule isn't machine-checkable — the batch transports auto-wire UseBenzeneInvocation inside a per-message sub-pipeline the check can't see — so that one stays a documented footgun.)

What reaches your logs per transport

When a handler throws, what Benzene logs (before any middleware you add) depends on the transport application that owns the invocation:

Two takeaways: the AWS batch family and the self-hosted workers already attribute a throw to a specific message; the single-event AWS Lambda sources (S3/EventBridge/Kafka) lean on the platform host, so wiring UseExceptionHandler/UseLogResult there is what gets you a Benzene-context log line instead of a bare stack trace in CloudWatch.

Checklist

When a message failed and you're trying to find out why:

1. Was it an unsuccessful result or a throw? Look for the router Warning (result) vs. an Error with a stack (throw).
2. Grep by traceId (or invocationId) to pull every line of that one message together.
3. Check the topic/handler tags to confirm it routed to the handler you expect — a "No handler found for topic" warning means a routing/registration problem, not a handler bug.
4. Open the trace if you export spans — the Error-status span names the failing stage.
5. Nothing in the logs at all? Confirm a logging provider is configured (not just AddLogging()), and that UseLogResult/UseExceptionHandler sit before .UseMessageHandlers().

Further reading

• Monitoring & Diagnostics — configuring logging providers, tracing, W3C context, and OpenTelemetry export
• Global Error Handling — UseExceptionHandler behavior and per-transport response mapping
• Common Middleware — reference for UseBenzeneEnrichment, UseLogResult, UseBenzeneMetrics
• Correlation Ids / Request Correlation cookbook — tracking a request across services
• Privacy & Data Handling — keeping PII out of the logs/traces above