Benzene

Capability Matrix — what Benzene does, deliberately doesn't, and how to fill the gap

Benzene is honest about its boundaries. This page is the single place that states, for each common production concern, what Benzene provides, what it deliberately does not do (and why), and how to solve the rest outside Benzene. Nothing here is a hidden gap — these are design decisions that follow directly from Benzene's philosophy.

The one idea behind every row

Benzene abstracts at the business-logic boundary — you write a message handler once and host it anywhere — and never at the transport or storage boundary. This is the opposite of frameworks like Dapr that hide the third party behind a generic interface (a "queue", a "state store") and, in doing so, hide that third party's best features. If you wrap SQS behind a generic queue interface, you lose the SQS-specific capabilities that were the reason to choose SQS.

So Benzene's answer to "does it abstract X?" is usually a deliberate no — you keep full, direct access to the underlying SDK and all its cloud-native capabilities, and Benzene stays out of the way. When Benzene doesn't ship an adapter for something, rolling your own is a first-class, supported path: a small piece of middleware or a custom pipe into the pipeline, not an escape hatch.

Two corollaries you'll see below:

The matrix

Capability What Benzene provides What it deliberately does NOT do (and why) How to solve the rest
Transport features Full, direct access to the underlying SDK message/context on every adapter. A send that fails in transport is a result, not an exception (transport-bindings §1, spec 1.1): every outbound route transport (UseSqs, UseSns, UseEventBridge, UseServiceBus, UseEventHub, UseEventGrid, UseQueueStorage, UsePubSub, UseKafka, UseRabbitMq, UseBenzeneMessageOverHttp; plus the plain REST UseHttp) yields a timeout result when the deadline elapsed (a TimeoutException in the chain — HttpClient.Timeout, an SDK's own timeout, a RabbitMQ confirm wait — or the route's own UseTimeout()) and service-unavailable otherwise, when the send fails before a reply or before the broker accepts the message (TransportFailure, mapped by each converter's ITransportFailureMapper in ContextConverterMiddleware); the IBenzeneMessageClient clients, StepFunctionsClient and the mesh dispatchers (HttpMeshServiceDispatcher, AwsLambdaMeshServiceDispatcher) classify the same way, and the Service Bus / Event Hubs batch clients report a batch that never reached the broker as every entry failed. Request serialization errors and the caller's own (ambient) cancellation still throw; the outbox relay reschedules a send that returns a failed result Hide transport-specific capabilities behind a generic interface (the anti-pattern above) Use the raw SDK feature directly — the context exposes the native message/event
Message routing Topic-based dispatch to [Message] handlers; the same handler runs behind every transport. .UsePresetTopic()/.UseTopicFrom() (fix a topic, or derive it per-message) work on every AWS Lambda trigger package with per-record topic routing — SQS, SNS, S3, DynamoDB Streams, Kafka, and EventBridge alike — plus the other non-HTTP transports listed in UseTopicFrom's own doc comment. On a metadata transport the topic key (topic by default) is read case-insensitively and is consumed as routing metadata, not surfaced again in the header dictionary (so forwarding a message's headers never re-routes it); Kafka routes on the record's topic header and falls back to the Kafka topic the record was consumed from, as RabbitMQ falls back to its routing key. EventBridge, which has no per-message attributes, carries wire headers inside detail under the reserved benzene-headers key (spec-0.2.0, wire-contracts §2 — _benzeneHeaders before it, no dual-accept), read into the header dictionary and left in the body, both directions Impose a canonical message envelope on transports that already carry routing (Kafka topic, Service Bus properties). Kinesis (Benzene.Aws.Lambda.Kinesis) has no preset/derived-topic support and never will by the same design: it fans a batch in to one StreamContext<KinesisEventRecord> with no per-record topic-routing concept at all, so there is no topic getter to wrap — not a gap, a consequence of the fan-in shape (see its own CLAUDE.md) For envelope-less transports (Queue Storage, Event Hubs bodies) use the Benzene envelope or a preset topic; otherwise route on the native key. For Kinesis, route within the stream handler itself (e.g. on PartitionKey)
Idempotency IIdempotencyStore seam + InMemoryIdempotencyStore (single-process) + atomic-claim middleware. Every claim is fenced: a win returns an opaque ClaimToken that CompleteAsync/ReleaseAsync require back, so a settle whose token no longer matches the live claim (reclaimed by another worker, or already settled) is refused (false, nothing written) rather than clobbering the current holder. IdempotencyMiddleware forwards the ambient CancellationToken (via ICancellationTokenAccessor) into every store call, matching the sibling Benzene.ClaimCheck middleware, so a wrapping UseTimeout/shutdown actually bounds a slow store call. IdempotencyMiddleware's outcome check now applies the SAME "null == failure, redeliver" convention every transport uses (SQS/DynamoDb always had it; SNS/S3/EventBridge got it in round 15's #229): a result-bearing (IHasMessageResult) context that completes without the pipeline ever setting MessageResult is treated as NOT proven successful and releases the claim, rather than being silently recorded Completed and permanently suppressing a real redelivery (round 16, #260 — the old fall-through disagreed with every transport it sits in front of). Fencing is token match alone across every implementation (InMemoryIdempotencyStore, DynamoDbIdempotencyStore, both Outbox stores) — a holder that merely outraces its own TTL, with nobody having reclaimed the key, can still settle with its original token. DynamoDbIdempotencyStore.TryClaimAsync never reports a win it didn't write: a conditional PutItem that loses to a live record which then reads back absent (a race with a concurrent release) is bounded-retried rather than synthesized as a win — every Won corresponds to an actual successful write. IdempotencyMiddleware's exception-path release is itself fenced against masking: if the handler throws and _store.ReleaseAsync then also throws (a real store failure, not a fenced false), that store failure is caught and logged, never replacing the original handler exception in the rethrow. Cross-instance de-duplication — independent processes can't coordinate at runtime without external state, and adding shared state can just relocate the race, not remove it An external store with an atomic conditional write (DynamoDB conditional put, Redis SET NX) keyed on message identity, plus handlers designed to be naturally idempotent. See Idempotency.
Outbox (produce-side durability/atomicity) UseOutbox() route middleware (Benzene.Outbox) with two write modes: Immediate (default) durably stores-and-forwards the send — survives process death/transport outages, retried with backoff, never silently swallowed, but not atomic with the handler's own state write; Transactional stages the envelope and defers the write to a store's own unit of work. Delivery is always at-least-once, never exactly-once, and has no ordering guarantee across envelopes. Every claim is fenced: ClaimDueAsync/ClaimAsync stamp a LeaseToken that MarkDispatchedAsync/RescheduleAsync/ParkAsync require back, closing the live-but-slow-claimant hole (a lapsed lease reclaimed by another worker can no longer have its settle clobbered by the original, stale claimant) — it does not and cannot recall a message a stale claimant already handed to the transport before its lease lapsed. OutboxDispatcher distinguishes a failed send from a settle-call failure after a genuinely successful send: only a SendAsync failure drives the reschedule/park path; a MarkDispatchedAsync throw (a routine transient store error, not the fenced false) is retried once and, if it still fails, the envelope is deliberately left claimed — outcome OutboxDispatchOutcome.SentButUnsettled — for the sweeper to reclaim once its lease lapses, rather than being treated as a failed send (which would guarantee a duplicate delivery). Benzene.Outbox.DynamoDb's IDynamoDbOutboxTransaction.CommitAsync builds its TransactWriteItems from a non-destructive peek of the staged envelopes and only drains the buffer after the call succeeds, so a thrown (not just oversized) transact-write leaves the staged envelopes in place for a retry rather than losing them. Benzene.Outbox.EntityFramework's three settle methods catch DbUpdateConcurrencyException (a reclaim landing between their own SELECT and SaveChanges) and return false, matching IOutboxStore's documented strictly-true/false contract rather than letting it escape as a raw exception. Make Transactional mode's atomic commit real on its own — that needs a store package. Exactly-once delivery, or any ordering guarantee across envelopes (both explicitly out of scope, not just unbuilt). Crash-after-send remains an inherent at-least-once window — fencing narrows it, it does not close it, and neither does the settle-retry above (it only closes the routine transient store error becoming a duplicate, not a genuine crash mid-settle). For the atomic commit, add Benzene.Outbox.DynamoDb (one TransactWriteItems via IDynamoDbOutboxTransaction) or Benzene.Outbox.EntityFramework (same-DbContext change tracker + your own SaveChangesAsync). At-least-once delivery means consumers must dedup — pair every outboxed route with Idempotency on the receiving end (the outbox stamps an idempotency-key header by default so the two packages click together with zero extra config). See Transactional Outbox.
Resilience UseRetry — retry with exponential backoff, an optional max-delay cap, and pluggable jitter (RetryMiddleware.FullJitter()), retrying a failed outbound send on its result with Benzene.Clients' OutboundRetry predicates (Transient, Unsuccessful, OnStatus) since a transport failure is a result, not an exception (register it before the transport, which ends the pipeline); UseTimeout — a deadline that becomes a timeout failure result while a genuine host cancellation still propagates, correctly classified even under nested UseTimeout composition regardless of which layer's timer actually fired; or the sequential-attempt subset of the Polly toolkit (retry, timeout, circuit breaker, rate limiting) via Benzene.Resilience.Polly's .UseResiliencePipeline(...), including a .ExcludingCancellation() PredicateBuilder helper so widening a strategy's ShouldHandle (e.g. a circuit breaker) doesn't silently drop Polly's own cancellation-safe default. Polly's per-attempt CancellationToken (Timeout/RateLimiter) reaches downstream code too — PollyResilienceMiddleware exposes it via the same ambient ICancellationTokenAccessor UseTimeout uses, linked with any outer ambient token so neither is lost; like UseTimeout, this only cancels work that actually reads the accessor's token. PollyResilienceMiddleware guards against re-entrancy: if a concurrent-attempt strategy (built via Polly's own AddStrategy(...) extensibility point — Hedging/Fallback can't even be built against the non-generic builder this package hands out) starts a second attempt while one is in flight, it throws NotSupportedException rather than running the downstream pipeline twice against the shared context/token (#267). Hedging and Fallback are not supported — they're generic-TResult-only in Polly.Core 8.5.0 (this package's builder is non-generic, since Benzene results flow through the mutable context, not a TResult), and, more fundamentally, this middleware shares one next continuation/context/ambient token across attempts, which only concurrent-attempt strategies (Hedging is the paradigm case) would violate — see Polly Resilience Pipelines. Benzene does not re-implement or hide Polly behind its own abstraction — the Polly package runs your ResiliencePipeline, exposing its full configuration surface for the strategies it does support For retry/timeout with zero extra dependency use UseRetry/UseTimeout; for anything more (within the sequential-attempt set) add Benzene.Resilience.Polly and drop a Polly ResiliencePipeline into .UseResiliencePipeline(...) — it even bridges a returned failure result to Polly's outcome model. For hedging-style concurrent calls, implement it at a different layer (around, not through, this middleware). See Polly Resilience Pipelines and Resilience.
Oversized payloads (transport size limits) Benzene.ClaimCheck — an optional offload/hydrate middleware pair: UseClaimCheck() offloads a payload over a configurable threshold to a pluggable IClaimCheckStore and carries a reference on the benzene-claim-check header; UseClaimCheck<TContext>() resolves it back before deserialization, forwarding the ambient CancellationToken (via ICancellationTokenAccessor) into every store call so a wrapping UseTimeout/shutdown actually bounds it. Store packages: Benzene.ClaimCheck.Aws.S3, Benzene.ClaimCheck.Azure.Blob. The built-in InMemoryClaimCheckStore reclaims an expired entry when it is next read, and separately sweeps all expired entries on PutAsync (at most once per minute) so a payload that is never read back at all still gets reclaimed, without a background thread. Compression (a smaller payload avoiding offload entirely), delete-on-consume (fan-out and at-least-once redelivery make it unsafe — see the doc), and interop with a third party's own offload wire format (e.g. the SQS Extended Client Library rides its reference in the body, not a header, so it isn't a drop-in swap) Retention is TTL-based, sized to exceed queue retention plus any DLQ redrive window (an infrastructure lifecycle rule, not created by the package). See Claim Check.
Sagas / workflows In-process, compensation-based saga with LIFO rollback (Benzene.Saga). A built Saga (SagaBuilder.Build()) is an immutable descriptor tree — no per-execution outcome is ever stored on its stages/steps — so the same built Saga instance is safe to call RunAsync() on concurrently, any number of times, without one run's outcome corrupting another's. A configured ISagaStateStore call that itself throws (mid-run, or recording the final outcome) never aborts the saga or replaces its real result: the store failure is caught, rollback for any already-completed stage still runs exactly as if the store had succeeded, and the genuine outcome (including a full Succeeded) is returned with SagaResult.StateStoreFailure populated — so a caller that retries on a thrown exception can never re-run a saga that already succeeded just because its completion wasn't durably recorded. SagaResult.Failures surfaces every step that failed within a stage (not just the first, additive alongside the existing Failure/FailureException, which still mirror Failures[0]) for the normal case of two steps in the same stage failing concurrently. Durable crash-resume — the saga is in-memory Func closures that can't be re-hydrated after a process dies; the state store records progress for observability, not recovery For crash-durable, long-running or human-in-the-loop workflows, use a real durable orchestrator: AWS Step Functions, Azure Durable Functions, Temporal, or similar. Benzene's Benzene.Clients.Aws.StepFunctions package can start a Step Functions execution from a handler, but running/resuming the workflow itself is the orchestrator's job, not Benzene's. See Sagas.
Schema evolution The Confluent wire-format codec (solid, tested) + an ISchemaRegistryClient seam Structural Avro/JSON backward-compatibility checking in-box — the shipped TextualSchemaCompatibilityChecker only accepts byte-identical schemas. SchemaRegistrySerializer.Deserialize also does not validate the framed schema id against the caller's requested Type — it's producer-side interop framing, not registry-driven consumer resolution, so a corrupted/misrouted payload deserializes as whatever type the caller asked for rather than being rejected Point at a real schema-registry server, or supply your own ISchemaCompatibilityChecker. See Schema Registry.
Database / state access (nothing — by design) Any database or state-store abstraction — a database is not a transport, so wrapping one would hide its capabilities (the core anti-pattern) Your handler uses its own SDK/ORM directly (CosmosClient, EF Core, Dapper, DynamoDBContext, …). Benzene delivers the event in; persistence is yours
Configuration & secrets Provider-agnostic ISecretStore (env vars, mounted files, composed, cached) + fail-fast startup validation Ship maintained cloud secret-store adapters (Key Vault / Secrets Manager / SSM) as packages Copy the ~30-line adapter from Secrets & Configuration and reference the cloud SDK yourself (a shipped adapter is a candidate for post-1.0)
Outbound HTTP Benzene.Clients.Http middleware over an HttpClient you supply. The envelope-over-HTTP readers (HttpBenzeneMessageClient, UseBenzeneMessageOverHttp, HttpBenzeneMessageHealthCheck, Benzene.Mesh.Dispatch's HttpMeshServiceDispatcher) read the response envelope from the body whatever the HTTP status, fall back to the reverse HTTP table only when the body is not an envelope, and treat an envelope without isSuccessful as unsuccessful (spec transport-bindings §1, wire-contracts §1.2). Error fidelity (core-concepts §12): every client that receives a problem document rebuilds the result's errors from it with field and code intact, in order, and attaches the document verbatim for result.GetProblem() (Benzene.Clients.Common.ProblemDocuments) — the envelope clients from a failure envelope's body (a Void caller included), the plain REST client (UseHttp/HttpContextConverter) from a non-success application/problem+json response, and the envelope-over-HTTP readers from a non-envelope application/problem+json failure (a host or gateway's own document). Inside an envelope the status stays the envelope's (envelope-first); outside one - the plain REST client, and the envelope-over-HTTP readers' non-envelope fallback - a problem document's benzeneStatus is the status, so an application-defined status survives the hop (wire-contracts §4.1, spec 1.1), and the reverse HTTP table applies to every other failure; the document's HTTP status member is never read; an error body of any other content type is not read as a problem document, and a body that does not parse leaves the status with no errors Manage HttpClient lifetime for you (no built-in IHttpClientFactory wiring on the low-level path) Register a typed/IHttpClientFactory client and pass it in; correlation/trace propagation is applied on the Benzene-message outbound path
Distributed tracing W3C traceparent/tracestate inbound extraction on HTTP and async transports (SQS, SNS, Kafka — AWS Lambda, Azure Functions, and the self-hosted worker — and Event Hub, Azure Functions and the self-hosted worker); OpenTelemetry, exporter-agnostic. The root "W3CTraceContext.Root" span and the per-stage handler span are both marked Error (with the exception attached) when downstream code throws, so a failed request is never indistinguishable from a successful one in a trace viewer. ICorrelationId/InboundCorrelationIdMiddleware bound and sanitize a caller-supplied correlation-id header (length-capped, control characters incl. \r/\n rejected) before it reaches a log scope or an outbound header, falling back to the self-generated id on rejection. Sampling/backend-specific configuration (Jaeger, Application Insights, etc.) is yours to wire — Benzene exports via the standard OTel API, it doesn't ship a backend .UseW3CTraceContext() as the first middleware on any pipeline continues a caller's trace instead of starting a new one per hop. Outbound: UseW3CTraceContext() forwards OpenTelemetry's Activity.Current; a meshed service forwards the mesh span instead with routing.UseMeshTraceContext() (Benzene.Mesh.Wire, mesh.md §3 / M2: the traceparent carries the span the TraceEvent records, and wins over UseW3CTraceContext() on the same route). See Monitoring & Diagnostics and the distributed tracing cookbook.
Retry-on-handler-failure-result Per-transport: some retry a returned IBenzeneResult failure (IsSuccessful == false) the same as an exception by default; some only retry it if you opt in; some can't retry it at all today. See the breakdown below — the default is not uniform across transports, and on several it's silently unsafe. A single cross-transport reliability abstraction — retry semantics are inherently transport-native (Lambda batch-item-failure vs. Service Bus abandon vs. EventBridge destinations), so Benzene surfaces each transport's own mechanism rather than inventing one Know your transport's default (below); opt in where an Options/AckMode knob exists; if none exists, have the handler throw instead of returning a failure result for anything that must be retried. Any handler that can be retried needs to be idempotent — see the Idempotency row above.
Multi-tenancy (nothing as a framework feature today) — (a candidate for post-1.0) Roll a tenant-resolver middleware + a scoped tenant holder (the documented Multi-Tenancy pattern)
AuthN / AuthZ OAuth2 bearer (JWT) validation with a strict algorithm allowlist + scope-based authorization (Benzene.Auth.OAuth2); signing keys are resolved via full OIDC discovery (Authority) or a bare JWKS document (JwksUri), both cached with refresh-on-unrecognized-kid (OAuth2ConfigurationManagerFactory/JwksOnlyConfigurationRetriever). OAuth2BearerOptions.Validate() fails fast at wire-up (not on the first request) on: a non-https Authority/JwksUri unless RequireHttpsMetadata is explicitly false; a null/whitespace/"*" entry in ValidIssuers/ValidAudiences; a ValidAlgorithms entry that is "none" or not one of the standard JWS signing algorithms (Microsoft.IdentityModel.Tokens.SecurityAlgorithms's HS/RS/ES/PS 256/384/512 constants); and a ClockSkew above OAuth2BearerOptions.MaxClockSkew (15 minutes). Benzene.Auth.Core's role/policy authorization (RequireRole/RequirePolicy/RequireAuthorization) reads roles via a single RoleClaims helper (ClaimTypes.Role/role/roles, the last also expanding a JSON-array value — Azure AD's app-roles shape) that deploy/Mesh/Benzene.Mesh.Host's MeshAuthGate shares rather than reimplementing, so both agree on what counts as a role Ship a policy-engine (OPA/Cedar) adapter (rate limiting is shipped separately — see the Rate limiting row, Benzene.RateLimiting) Add an authorization middleware calling your policy engine; the Benzene.Auth.Core seams are there to build on
AI tool access (MCP) Benzene.Mcp: the Model Context Protocol as a transport binding — tools (topic-bound, or a body you write that asks the pipeline) over Streamable HTTP (default /benzene/mcp) on any Benzene HTTP host or stdio on the worker, a topic-bound call carrying benzene-mcp-tool and the session benzene-mcp-session (spec-0.2.0); a tool call goes through the same middleware, validation, mesh feeds and start-up checks as every other transport; strict argument reading and a start-up check that every tool's topic has a handler (docs) Generate one tool per handler or auto-expose every topic (a hundred schemas in front of a model is a worse interface than none); keep session state or issue OAuth tokens; offer the server-to-client stream, resources or prompts Design the tools as the jobs a model is asked to do; compose your auth middleware in front of the endpoint; read McpToolCatalog/McpRequestInfo in that middleware to gate the tools that write

Retry-on-handler-failure-result — the per-transport breakdown

"Failure result" below means your handler returned IsSuccessful == false (e.g. BenzeneResult.ServiceUnavailable(...)) — not a thrown exception. Every transport in Benzene retries an unhandled exception by default (that's what makes it propagate to the host's own retry machinery); what varies is what happens to a returned failure result, which is the surprising case because nothing crashed.

Transport Default on a returned failure result Opt-in fix
AWS Lambda SQS (Benzene.Aws.Lambda.Sqs) Safe — retried per-message via ReportBatchItemFailures, same as an exception. A null/unrouted outcome (no result recorded) is retained the same way, not silently deleted. SqsOptions.BatchFailureMode = FailWholeBatch to change the shape of the retry (the whole batch); SqsBatchFailureMode.AtMostOnce for at-most-once — every message acknowledged whatever its outcome, failures logged
AWS DynamoDB Streams (Benzene.Aws.Lambda.DynamoDb) Safe — sequential processing stops at the first failed record and reports it for redelivery. A null/unrouted outcome (no result recorded) stops the sequence and is reported the same way, not silently skipped. Unit of settlement: the first failed record's sequence number — it and every record after it in the batch are redelivered. UseDynamoDb(action, options => options.AtMostOnce = true) for at-most-once — a failed record is logged and skipped, the rest of the batch still runs in order
AWS Kinesis (Benzene.Aws.Lambda.Kinesis) Safe by design — the stream checkpoints only where your handler explicitly checkpoints; an unfailed-but-uncheckpointed record is redelivered. The checkpointer tracks a contiguous-prefix watermark (round 17, #273), not a single monotonic max-index one: the resume point is the first unconfirmed record's position, so a handler using StreamOperators.PartitionBy (this package's own recommended pattern for restoring per-key ordering) can never have an earlier, still-failed record silently skipped just because a later record from a different partition group was checkpointed first. Accepted tradeoff: a record confirmed ahead of an earlier gap is redelivered even though it already succeeded (safe over-retry, at-least-once) — for a plain sequential handler with no gaps, behavior is byte-identical to the old watermark KinesisStreamOptions.AtMostOnce = true for at-most-once — a throwing pipeline checkpoints the whole batch, so nothing is redelivered. Unit of settlement: the batch from the resume point on (fan-in: no per-record result) — see Benzene.Aws.Lambda.Kinesis's CLAUDE.md
AWS SNS (Benzene.Aws.Lambda.Sns) Safe by default — SnsOptions.RaiseOnFailureStatus defaults true, escalating a failure result into a throw so SNS's subscription retry/redrive applies. A null/unrouted outcome (no result recorded — typically an unrouted message) is escalated the same way, not silently accepted (flipped 2026-08-25 — see work/settlement-consistency-fix-plan.md). SnsOptions.CatchExceptions = true still can't swallow an infrastructure/DI-wiring failure (BenzeneResolutionException) — it rethrows unconditionally, because SNS has no partial-failure channel, so swallowing it would report every message healthy while all of them fail the same way. SnsOptions.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once (a failure result, no result and a handler exception all acknowledged) — see SNS Fan-Out Pattern
AWS EventBridge (Benzene.Aws.Lambda.EventBridge) Safe by default — EventBridgeOptions.RaiseOnFailureStatus defaults true, escalating a failure result into a throw so EventBridge's retry/DLQ applies. A null/unrouted outcome (no result recorded — typically an unrouted event) is escalated the same way, not silently accepted (flipped 2026-08-25 — see work/settlement-consistency-fix-plan.md). EventBridgeOptions.CatchExceptions = true still can't swallow an infrastructure/DI-wiring failure — it rethrows unconditionally, the same carve-out as SNS/S3. EventBridgeOptions.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once
AWS Kafka/MSK (Benzene.Aws.Lambda.Kafka) Safe by default — KafkaOptions.BatchFailureMode defaults to PartialBatchFailure, reporting each failed partition's resume offset via the batch response (needs ReportBatchItemFailures on the event source mapping, as with SQS) so records are redriven from the failure, honouring per-partition ordering. A null/unrouted outcome (no result recorded — a middleware ended the pipeline without setting one) is the deliberate exception: it is NOT escalated, it settles as if processed — skip-and-continue, the stream allowance of transport-bindings §1 obligation 8 — Kafka has no per-record dead-letter path, so retaining it would replay the partition from that offset forever (carve-out, unchanged — see work/settlement-consistency-fix-plan.md). An unmatched topic is not this case: the router records not-found for it, a failure, so it stops its partition like any other failure — use UseUnmatchedTopicAsIgnored() on a shared topic. Unit of settlement: per topic-partition, the first failed record's offset — it and every later record in that partition are redriven. KafkaOptions.BatchFailureMode = FailWholeBatch (throws) when the event source mapping isn't configured for ReportBatchItemFailures; KafkaBatchFailureMode.AtMostOnce for at-most-once — a failed record is logged and skipped, its partition carries on
AWS S3 (Benzene.Aws.Lambda.S3) Safe by default — S3Options.RaiseOnFailureStatus defaults true, escalating a failure result into a throw so the Lambda invocation fails and the event's async retry/DLQ applies. A null/unrouted outcome (no result recorded — typically an unrouted record) is escalated the same way, not silently accepted (flipped 2026-08-25 — see work/settlement-consistency-fix-plan.md). S3Options.CatchExceptions = true still can't swallow an infrastructure/DI-wiring failure — it rethrows unconditionally, the same carve-out as SNS/EventBridge. S3Options.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once
AWS SQS self-hosted worker (Benzene.Aws.Sqs's SqsConsumer) Safe by default — SqsConsumerOptions.AckMode defaults to SqsConsumerAckMode.PerMessage, so only successfully-handled messages are deleted and a failure result is left for redelivery. A null/unrouted outcome (no result recorded) is left for redelivery the same way, not silently deleted. SqsConsumerAckMode.AtMostOnce for at-most-once (delete every message whatever its outcome, a thrown exception included; failures logged). WholeBatch deletes the whole batch after a failure result but keeps it all on the queue if a handler throws
RabbitMQ self-hosted worker (Benzene.RabbitMq) Safe by default (RabbitMqAckMode.Explicit) — a failure result (or exception) nacks the message for requeue/dead-letter, not ack. A null/unrouted outcome (no result recorded — typically an unrouted delivery) nacks the same way, not accepted as success (flipped 2026-08-25, overturning the previously documented ack-on-null behaviour — see work/settlement-consistency-fix-plan.md). N/A (default is already safe); RequeueOnFailure/AckMode tune how it's redelivered, or AckMode = AutoAck to opt into at-most-once
Azure Service Bus Function trigger (Benzene.Azure.Function.ServiceBus) Safe by default — although AckMode defaults to AutoComplete, ServiceBusOptions.RaiseOnFailureStatus defaults true, so a failure result throws (→ the message is abandoned, not completed). Under AckMode = Explicit, state.Acked is set only after the CompleteMessageAsync/AbandonMessageAsync settle call itself returns successfully (ServiceBusApplication.OnPipelineSucceededAsync) — a throwing settle call is exactly the case the base class's exception-recovery fallback-abandon exists for, and setting Acked up front used to skip it. A null/unrouted outcome (no result recorded — typically an unrouted message) is treated the same as a failure on both ack paths: under AutoComplete it's escalated the same way (flipped 2026-08-25); under Explicit it was already abandoned, not completed (see work/settlement-consistency-fix-plan.md). Like every sibling sharing AzureFunctionBatchApplicationBase, CatchExceptions = true still can't swallow an infrastructure/DI-wiring failure (BenzeneFailure.IsInfrastructure) — it rethrows unconditionally after logging (#257 fix, mirroring AWS SNS/S3/EventBridge's #228), composing with the Explicit-ack abandon hook (abandon still happens, then the invocation fails loudly); the same catch also lets a genuine ambient OperationCanceledException (this call's own cancellation token, not an unrelated one) escape containment rather than being logged as an ordinary per-item failure (#258 fix), consistent with an already-queued item's #230 abort behavior. RaiseOnFailureStatus = false with CatchExceptions = true (and AckMode = AutoComplete, the default) for at-most-once; AckMode = Explicit for true per-message complete/abandon control
Azure Service Bus self-hosted worker (Benzene.Azure.ServiceBus) Safe by default — BenzeneServiceBusConfig.AckMode defaults to ServiceBusConsumerAckMode.Explicit, so a failure result (or throw) abandons the message for redelivery. A null/unrouted outcome (no result recorded) abandons the same way, not silently completed. Under AckMode = Explicit, settlement (SettleAsync — complete/abandon/dead-letter/defer) now runs after and outside the handler's own try/catch (#277): a settlement failure following a successfully-processed message (e.g. the lock was lost by the time settlement runs) is logged as a distinct settlement failure naming the message id, not misattributed to the handler, and does not trigger an abandon on a message that was already correctly processed — the lock is left to expire naturally so Service Bus redelivers it. A settlement failure for a message whose handler itself had failed keeps the prior log-and-propagate behavior unchanged. AckMode = ServiceBusConsumerAckMode.AtMostOnce for at-most-once — every message completed whatever its outcome, a thrown exception included (logged); a settlement the handler requested explicitly still wins. AutoComplete completes after a failure result but abandons a thrown exception
Azure Kafka/Event Hubs Function trigger (Benzene.Azure.Function.Kafka) Safe by default — KafkaOptions.RaiseOnFailureStatus defaults true, escalating a failure result into a throw so the trigger re-delivers. A null/unrouted outcome (no result recorded — a middleware ended the pipeline without setting one) is the deliberate exception: it is NOT escalated, it settles as if processed — skip-and-continue, the stream allowance of transport-bindings §1 obligation 8 — no per-record dead-letter path, so retaining it would replay the whole triggered batch forever (carve-out via EscalateUnestablishedOutcome => false — see work/settlement-consistency-fix-plan.md). An unmatched topic is not this case: the router records not-found, a failure, which escalates. Unit of settlement: the triggered batch. Shares AzureFunctionBatchApplicationBase's #257/#258 fixes above: CatchExceptions = true rethrows an infrastructure/DI-wiring failure and a genuine ambient cancellation instead of swallowing either. KafkaOptions.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once
Azure Event Grid (Benzene.Azure.Function.EventGrid) Safe by default — EventGridOptions.RaiseOnFailureStatus defaults true, escalating a failure result into a throw so Event Grid's retry/dead-letter (backoff up to 24h) applies. A null/unrouted outcome (no result recorded — typically an unrouted event) is escalated the same way, not silently accepted (flipped 2026-08-25 — see work/settlement-consistency-fix-plan.md). A malformed delivery (EventGridTriggerEvent.Parse throwing on invalid JSON) is now the same kind of per-event failure too — the parse is deferred to inside the pipeline's own guarded execution (EventGridContext's raw-JSON constructor), not run eagerly before dispatch, so it's caught/logged under CatchExceptions = true or left to cascade (Event Grid's retry/dead-letter applies) under the default false, rather than an unguarded crash regardless of either option (round 14-15, #235). Shares AzureFunctionBatchApplicationBase's #257/#258 fixes above: CatchExceptions = true rethrows an infrastructure/DI-wiring failure and a genuine ambient cancellation instead of swallowing either. EventGridOptions.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once
Azure Event Hubs Function trigger (Benzene.Azure.Function.EventHub) Safe by default — EventHubOptions.RaiseOnFailureStatus defaults true, escalating a failure result (on both the property-based and envelope routing paths) into a throw so the trigger re-delivers the batch. A null/unrouted outcome (no result recorded — a middleware ended the pipeline without setting one) is the deliberate exception: it is NOT escalated, it settles as if processed — skip-and-continue, the stream allowance of transport-bindings §1 obligation 8 — no per-record dead-letter path, so retaining it would replay the whole triggered batch forever (carve-out via EscalateUnestablishedOutcome => false — see work/settlement-consistency-fix-plan.md). An unmatched topic is not this case: the router records not-found, a failure, which escalates. Unit of settlement: the triggered batch. Shares AzureFunctionBatchApplicationBase's #257/#258 fixes above: CatchExceptions = true rethrows an infrastructure/DI-wiring failure and a genuine ambient cancellation instead of swallowing either. EventHubOptions.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once
Azure Queue Storage (Benzene.Azure.Function.QueueStorage) Safe by default — QueueStorageOptions.RaiseOnFailureStatus defaults true, escalating a failure result (on both the preset-topic and envelope routing paths) into a throw so the host's maxDequeueCount retry/poison handling applies. A null/unrouted outcome (no result recorded — typically an unrouted message) is escalated the same way, not silently deleted (flipped 2026-08-25 — see work/settlement-consistency-fix-plan.md). Shares AzureFunctionBatchApplicationBase's #257/#258 fixes above: CatchExceptions = true rethrows an infrastructure/DI-wiring failure and a genuine ambient cancellation instead of swallowing either. QueueStorageOptions.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once
Azure Timer trigger (Benzene.Azure.Function.Timer) Visible by default, not redeliverable — TimerOptions.RaiseOnFailureStatus defaults true, escalating a failure result into a thrown TimerMessageProcessingException (#231 fix) so the Functions host records a failed invocation (retry/alerting per your monitoring, not Benzene) instead of completing silently; unlike every queue-shaped transport above, a failed tick is never redelivered by the platform either way — the next occurrence just runs on schedule, so the flag only controls whether the failure is visible, not whether it's retried. TimerOptions.CatchExceptions = true still can't swallow an infrastructure/DI-wiring failure — it rethrows unconditionally after logging (#257 fix, independent of but mirroring the batch-trigger family's) TimerOptions.RaiseOnFailureStatus = false to accept a failure result without raising; TimerOptions.CatchExceptions = true to also contain a thrown exception instead of failing the invocation
Google Cloud Pub/Sub (Benzene.GoogleCloud.Functions.PubSub) Safe by default — PubSubOptions.RaiseOnFailureStatus defaults true, escalating a failure result into a throw so Pub/Sub redelivers per the subscription's ack-deadline/retry policy. A null/unrouted outcome (no result recorded — typically an unrouted message) is escalated the same way, not silently accepted (flipped 2026-08-25 — see work/settlement-consistency-fix-plan.md). PubSubOptions.RaiseOnFailureStatus = false with CatchExceptions = true for at-most-once
Kafka self-hosted worker (Benzene.Kafka.Core) At-most-once by default (skip-and-continue) — a stream worker (see the callout below) can't redeliver one record without halting, so BenzeneKafkaConfig.CommitOnlyOnSuccess defaults false: offsets auto-commit regardless of outcome, so a failed record is skipped on restart (a returned failure result is at least logged — RaiseOnFailureStatus defaults true). Unit of settlement: the record's offset, committed per partition. A null/unrouted outcome (no result recorded — a middleware ended the pipeline without setting one) is the deliberate exception: it is NOT escalated, it settles as if processed (ack'd) — no per-record dead-letter path, so retaining it would replay the partition from that offset forever (carve-out, unchanged — see work/settlement-consistency-fix-plan.md). BenzeneKafkaConfig.CommitOnlyOnSuccess = true (requires CatchHandlerExceptions = false) for at-least-once — offsets are then stored only after a successful handle, so a restart redelivers the failed record; or wire a DeadLetterTopic. Under either, RaiseOnFailureStatus (default true) settles a returned failure result the same way as a thrown one; set it false to accept a failure result as settled
Azure Event Hubs self-hosted worker (Benzene.Azure.EventHub) At-most-once by default (skip-and-continue) — same stream constraint: BenzeneEventHubConfig.CatchHandlerExceptions defaults true, so a failed event (or an escalated failure result — RaiseOnFailureStatus also defaults true) is logged and skipped once a later event checkpoints past it. Unit of settlement: the partition checkpoint, every CheckpointInterval events. A null/unrouted outcome (no result recorded — a middleware ended the pipeline without setting one) is the deliberate exception: it is NOT escalated, it settles as if processed (ack'd) — no per-record dead-letter path, so retaining it would replay the partition forever (carve-out, unchanged — see work/settlement-consistency-fix-plan.md). BenzeneEventHubConfig.CatchHandlerExceptions = false for at-least-once — the worker then stops at the failure without checkpointing, so a restart redelivers (and RaiseOnFailureStatus = false additionally accepts a failure result as settled)
Azure Cosmos DB Change Feed — Functions trigger (Benzene.Azure.Function.CosmosDb) N/A as a "failure result" concern — like Kinesis, this is a fan-in StreamContext<TDocument> with no IBenzeneResult/message-handler routing; the trigger's lease checkpoints the whole batch on any non-throwing return, and unlike Kinesis there's no per-document checkpoint API to opt out of that with (see the package's own CLAUDE.md). The Blob trigger (Benzene.Azure.Function.BlobStorage) is the other deliberately non-routed Azure adapter — it delivers a blob's name + content to UseBlob(...) rather than routing a message, with the same throw-to-redeliver escape hatch. The [assembly: BenzeneCosmosDbTrigger(...)] source-generator reader now validates DatabaseName/ContainerName the same way every sibling transport's own destination field is validated (BENZ0010, #259 fix) — a declaration missing either reports a build-time error and emits nothing, alongside (not instead of) the pre-existing DocumentType/BENZ0002 check, instead of compiling clean and shipping a CosmosDBTrigger(databaseName: "", containerName: "", ...) that only fails at Azure host startup Have the handler throw for anything that must redeliver the whole batch (unit of settlement: the triggered batch); UseCosmosDbChangeFeed<T>(action, options => options.AtMostOnce = true) for at-most-once — a throwing batch is logged and the trigger checkpoints past it. The Blob trigger (fixture binding blob-storage, one-shot) settles the same way — a throw fails the invocation for the host's retry and poison queue; UseBlobStorage(action, options => options.AtMostOnce = true) is its at-most-once override
Azure Cosmos DB Change Feed — self-hosted worker (Benzene.Azure.CosmosDb) N/A as a "failure result" concern, same fan-in StreamContext<TDocument> shape as the Functions trigger above, but with manual checkpoint control the trigger doesn't expose. BenzeneCosmosChangeFeedWorker's BenzeneCosmosChangeFeedConfig.AutoCheckpointOnSuccess defaults true (checkpoints automatically once the pipeline completes, unless the handler already checkpointed itself via the batch's own checkpointer) and CatchHandlerExceptions defaults false (a failed batch propagates uncheckpointed, so the SDK's Change Feed Processor redelivers it — at-least-once by default, the opposite default from the Event Hubs self-hosted worker above, because the change feed natively retries a failed batch and Event Hubs doesn't); its sibling BenzeneCosmosAllVersionsChangeFeedWorker (all-versions-and-deletes mode) is automatic-checkpoint-only — no AutoCheckpointOnSuccess knob exists, only BenzeneCosmosAllVersionsChangeFeedConfig.CatchHandlerExceptions (same default/semantics). On BenzeneCosmosChangeFeedWorker only, the auto-checkpoint call now runs after and outside the handler's own try/catch (round-10 #108): a lease-container write failure following a successfully-processed batch is logged as a checkpoint failure (naming the lease token, not the handler) and left un-checkpointed for redelivery, never misattributed to the handler and never retried a second time with zero backoff. The skip-mode ("checkpoint the failed batch anyway so it's permanently passed over") checkpoint call is now also wrapped in its own try/catch (#276): a lease-container failure there (e.g. a 429 throttle) is logged with a distinct message naming the lease container and skip mode, then swallowed, instead of escaping OnChangesAsync entirely unhandled and unlogged — the batch is simply left un-checkpointed and retried on the next partition scan CatchHandlerExceptions = true on either worker for skip-and-continue (permanently passes over a poison batch); AutoCheckpointOnSuccess = false on BenzeneCosmosChangeFeedWorker for fully manual checkpoint control via the batch's own checkpointer
HTTP / API Gateway N/A — a failure result maps straight to an HTTP status code the caller sees synchronously; there's no async "retry" concept to opt into N/A

The 1.0 settlement contract (maintainer-approved 2026-07-21): every queue-shaped transport is safe by default — a handler that returns a failure IBenzeneResult (rather than throwing) is not settled by default; it is escalated as that transport's *MessageProcessingException so the platform redelivers it (at-least-once), instead of being silently acked/completed/checkpointed past (at-most-once). Mechanically, RaiseOnFailureStatus defaults true on every queue-shaped options type (AWS Lambda SNS/S3/EventBridge; the Azure Functions Kafka, Queue Storage, Event Grid, Event Hub and Service Bus triggers; the self-hosted Event Hub worker; Google Cloud Pub/Sub), while Benzene.Aws.Sqs (AckMode = PerMessage), Benzene.Azure.ServiceBus and Benzene.RabbitMq (AckMode = Explicit) were already safe, and Kinesis / DynamoDB Streams settle via ReportBatchItemFailures instead. Each transport exposes an explicit opt-out (the "Opt-in fix" column shows the reverse) for the at-most-once cases where you deliberately want a returned failure accepted. SettlementContractDefaultsTest guards these defaults against drift; the migration record behind this contract is archived at work/archive/settlement-contract-1.0-2026-07.md.

A second, separate axis (decided 2026-08-25): a null/unestablished outcome — no result recorded at all, overwhelmingly an unrouted message (no handler matched the topic) — is not success either. It is retained/redelivered wherever a redelivery backstop (DLQ, poison queue, dequeue-count, subscription retry/redrive) exists to catch it, exactly like a returned failure result above; it is accepted (ack'd/ committed) only on the five adapters with no per-record dead-letter path, where retaining it would replay a Kafka partition or Event Hub batch forever — Benzene.Aws.Lambda.Kafka, Benzene.Azure.Function.Kafka, Benzene.Kafka.Core, Benzene.Azure.Function.EventHub, and Benzene.Azure.EventHub, each marked above. Kinesis and Cosmos DB Change Feed have no per-record result at all (fan-in — see their own rows) so this axis doesn't apply to them. SettlementContractDefaultsTest.NullOutcomePolicy_MatchesTheDecidedTable guards this axis against drift the same way; the full per-adapter table and reasoning are in work/settlement-consistency-fix-plan.md.

Spec 0.2 — settlement is a cross-port obligation (transport-bindings.md §1, obligation 8, in the benzene repo; pinned by transport-metadata-cases.json → settlementDefaults). Three things it adds to the axes above, each now true of every one-way binding here:

Each binding's shape (queue, one-shot, stream, request-response) and defaults are asserted against the fixture by SettlementDefaultsConformanceTest — including that every stream default of skip-and-continue says so in its row above. The Azure Functions Timer and Blob triggers are one-shot bindings (timer, blob-storage); there is no asynchronous envelope entry point (envelope-async): the Lambda direct-invoke envelope handler is request/response only.

The two self-hosted stream workers are the deliberate exception — they default to at-most-once. A stream (a Kafka partition, an Event Hub partition) has no per-message ack/abandon: the only way to not lose a failed record is to stop the worker and never advance the offset/checkpoint past it. That is far too drastic a default (one poison record would halt all processing), so Benzene.Kafka.Core (CommitOnlyOnSuccess = false) and Benzene.Azure.EventHub (CatchHandlerExceptions = true) default to skip-and-continue — a failed record is logged and the stream advances past it. Opt into at-least-once with CommitOnlyOnSuccess = true / CatchHandlerExceptions = false respectively, and accept that a poison record then wedges/halts the worker until it's handled or dead-lettered. This is a genuine property of streams, not an oversight — the Lambda/Functions stream triggers (Kinesis, DynamoDB Streams, the Kafka/Event Hub Function triggers) can be safe-by-default because the platform re-invokes them from the un-advanced checkpoint without any single process having to halt; a long-running self-hosted worker has no such external re-invoker.

The rows marked N/A above (Cosmos DB Change Feed, HTTP) have no per-item async-retry knob to opt into — there, a handler that throws is the escape hatch for anything that must redeliver.

Area rows — packages and surfaces

The rows above are organised by production concern; the rows below are organised by package or area, in the shape the sibling ports' matrices use, so the cross-port matrix can read this port row-for-row. Same rules: every "provides" claim names the code that provides it, a deliberate "no" states its reason, and anything simply unbuilt says "not implemented" plainly.

Area What it provides What it deliberately does NOT do (and why) How to solve the rest
gRPC Benzene.Grpc — routes all four RPC shapes (unary, server-streaming, client-streaming, bidirectional; BenzeneInterceptor overrides each) into message handlers via [GrpcMethod], with protobuf⇄JSON bridging and status-code mapping; a server-streaming/bidirectional handler exception thrown mid-stream (after the pipeline itself already succeeded) is classified the same way a unary handler's exception is (ArgumentException → ValidationError, TimeoutException → Timeout, a genuine OperationCanceledException → Cancelled/DeadlineExceeded, else ServiceUnavailable), with the benzene-status trailer written only once the stream has fully drained so a mid-stream failure can never leave a stale ok trailer on a call that actually failed; Benzene.Grpc.AspNet — ASP.NET Core hosting with opt-in grpc.health.v1 (including named liveness/readiness services, each an aggregate BenzeneHealthCheckBridge that fails fast at construction — not at first probe — if a configured LivenessCheckTypes/ReadinessCheckTypes entry matches no registered check, suffixes duplicate Type keys in its reported data rather than letting one collide over another, and applies HealthCheckProcessor.RunTimedAsync's same non-critical-failure downgrade — Failed && IsNonCritical && !IsPersistent reports as degraded, not unhealthy — so a non-critical dependency (including the always-non-critical DependencyHealthCheck category) never disagrees between the HTTP/message-handler health path and grpc.health.v1) and reflection; Benzene.Grpc.Client — outbound unary client with inbound-deadline propagation, classifying ambient send-time cancellation (OperationCanceledException) as a ServiceUnavailable result without Error-level logging, same as a mid-flight RpcException(Cancelled). Structured errors (wire-contracts §4.2, spec 1.1): a failure's errors travel as google.rpc.BadRequest field violations whatever its status, each error's code in FieldViolation.reason (GrpcFieldViolationReason, field 3 on the wire), and the client rebuilds message/field/code from them; Benzene.Grpc.Versioning — request-side payload-version casting; Benzene.Grpc.TestHelpers Own the gRPC server or generate service stubs — the interceptor claims only routed methods; unmatched calls fall through untouched to the native service implementation. Response-side payload downcasting over gRPC — the result setter writes protobuf directly, with no response payload mapper to decorate (see Benzene.Grpc.Versioning's CLAUDE.md) Register real protoc-generated services alongside; model an older response shape as a distinct method/topic
Mesh — service side Benzene.Mesh.Wire — the mesh.md wire layer. spec-0.4.0 (Phase 3b): the register body is the contract document — a service's only self-description is its contract document at benzene:spec (Benzene.Schema.OpenApi, row below), and it registers that same document, with runtime/placement/profile/degraded, on benzene:mesh:register; the ServiceDescriptor, descriptorHash, the reserved benzene:mesh topic (UseMeshDescriptor) and MeshSchemaGenerator are gone. Heartbeats (MeshHeartbeat) carry serviceVersion and the registered document's registrationHash. Send declarations: MeshOutboundRegistry (AddSendDeclarations / the cloud service's WithSends), and a generated client declares the topics it calls as sends of its consumer (every registered IMessageDefinitionFinder<IMessageSenderDefinition> reaches the document; a (topic, version) declared twice is listed once). Trace middleware with W3C traceparent join (the joined tracestate carried on MeshSpan.TraceState) and outbound forwarding of the mesh span — UseMeshTraceContext(), per route or on every route through the OutboundRoutingBuilder (sends and published events), stamps MeshSpan.Current's traceparent/tracestate and pins it over UseW3CTraceContext()'s Activity.Current, so a downstream TraceEvent's parentSpanId is the upstream TraceEvent's spanId (mesh.md §3, M2); the Cloud Service's M2 self-check is satisfied only when the routing forwards it (or there are no outbound routes) — + the batching HTTP trace exporter — HttpMeshTraceExporter tracks an absolute next-flush deadline, reset only after an actual flush, so a steady trickle below batchSize still gets a periodic time-based flush rather than only flushing at process shutdown (round 15, #233) — and the deduplicating issue emitter (IMeshIssueExporter) — whose issues carry error detail (spec 1.2, mesh §4.1): UseMeshIssues takes each failing result's structured errors as (field, code) pairs, never the message (an error with neither, or empty ones, is not counted), and HttpMeshIssueExporter counts them per issue as deltas, caps the distinct pairs a window holds at 100 and bounds the flushed errors to the 10 highest counts (then field, then code, ordinal, absent first); MeshRegistration is the collector's view of a register body; Benzene.Mesh.Reporting — push self-report for services nothing can poll (SQS-only Lambdas etc.). Passes the contract-document producer groups (its document is the register body) and mesh-trace-cases.json Turn any feed on by default — every mesh feed is opt-in and independent, and a service is fully functional without the mesh; forward the mesh span (MeshSpan.Current) on an outbound route the application did not wire — UseMeshTraceContext() is opt-in per route or on every route through OutboundRoutingBuilder, and the cloud service does not own the application's routes (the M2 self-check reports whether every route forwards it) UseBenzeneCloudService (row below) wires benzene:spec, the trace feed, registration of that document and heartbeats in one call — the meshed-service claim's M1 (mesh §7.1)
Mesh — collector Benzene.Mesh.Collector — spec-0.4.0 (Phase 3b): a register body is a contract document (MeshRegistration) of which only service is required (no dialect check; every other member may be absent, and unmodelled members are kept); consumers come from requests, providers from events ∪ sends (a published event makes its publisher a provider), benzene: entries are ignored, and a member that is absent or named in degraded is unknown — it contributes no edges and a trace on it is never filed as undeclared; a trace of a reserved benzene: topic is never drift; the carried registrationHash is trusted (never recomputed, never a reason to reject) and a heartbeat's hash is compared with the registration for the same (service, serviceVersion) — before, with no version on the heartbeat, it matched any registered version — and is null (unknown) for a version nothing registered; the query shapes stay Provisional and fixture-compatible (topics on the fleet row counts requests; missingFeeds' descriptor means "never registered"), with ServiceView.descriptor renamed registration and InstanceView.descriptorHash renamed registrationHash (+ serviceVersion); no legacy topics/produces body is read. Passes the re-cut mesh-collector-cases.json and mesh-service-version-cases.json. benzene:mesh:register/heartbeat/traces/issues ingest (spec 1.2: an issue's errors decode leniently — a non-object entry, one with neither field nor code, or a count that is not a positive integer is skipped, never failing the batch — and merge by (field, code) under the same bound and order; a fleet trace summary carries the flow's status, its earliest failing event's when it failed, else its earliest event's, set by the push store and the Jaeger and enriched X-Ray rows, omitted by Tempo and X-Ray summary-plane rows, which hold no Benzene status) plus benzene:mesh:query:{fleet,service,topic,trace,correlation} read models — all five query handlers resolve the ambient ICancellationTokenAccessor at the point of use and thread the token into every read-model call, the same idiom MeshDispatchMessageHandler uses, so UseTimeout(...) wrapping the query envelope genuinely bounds a slow/backend-composed read (e.g. an abandoned browser query against a trace-backed plane) instead of running the full backend scan to completion regardless of the caller having gone away (round 16, #250) — over the in-memory MeshCollectorStore (bounded trace ring; the catalog key is (service, serviceVersion) per spec §2.4, claimed against mesh-service-version-cases.json — the fleet query keeps one row per service, headlined by the most recently registered version, and the service query's versions lists every live version as {serviceVersion, topics}, oldest registration first (D26, spec-0.2.0) — Registering a second live version of a service never evicts a still-live sibling version's registration from the catalog, and an instance's HashMatches is compared against its own version's registration, so two versions reporting different-but-individually-correct hashes (a canary/blue-green rollout) read as healthy side by side instead of false contract drift, while a genuine same-version re-registration with a changed hash still reads as drift (round 16, #251); the producer/consumer graph is declared by registrations, never trace-derived; Register/AddEvents/AddIssues null-coalesce every wire-supplied list — Topics/Produces, an events batch, an issues batch, and each issue's exemplar trace ids — to empty rather than throwing NullReferenceException on an explicit-null list, honoring the spec's "no missing feed ever fails ingestion" contract all the way down to whole collections, not just individual nullable fields (round 15, #234) — and, one level down, AddEvents/AddIssues also skip a null ELEMENT inside a non-null list (legal on the wire: MeshTraceEvent/MeshIssue are reference types) rather than dereferencing it, so one malformed item can no longer partially corrupt a batch and drop every item ingested after it (round 16, #253)); the traffic-observed contract-drift signal (RecordObservedActivityAndDrift) now reasons about live versions the same way HashMatches does — a topic counts as declared when ANY currently live version's requests (or events ∪ sends) lists it, not just the most-recently-registered ("headline") version, so an older-but-still-live version handling traffic on an edge only it declares is no longer misfiled as contract drift the moment a newer sibling version registers (round 17, #283 — a documented approximation: MeshTraceEvent carries no per-event ServiceVersion, so a genuine single-version drift on an edge another live version also happens to declare goes undetected until that other version retires; open wire-shape question in work/outstanding-bugs.md); ServiceState.Descriptors is now capped at a configurable maxVersionsPerService (default 8, a constructor parameter alongside maxTraceEvents/maxIssues) instead of retaining one permanent entry per historical ServiceVersion for the life of the process — registering a version past the cap evicts the least-recently-registered non-current version (preferring one with no live instance currently reporting its hash; the cap wins over liveness if every retained version has one) and retracts its edges via the same RetractEdges path a same-key re-registration already uses, so nothing is left dangling in the topic catalog (round 17, #290; cap-vs-TTL policy choice recorded as open in work/outstanding-bugs.md); CompositeMeshFleetReadModel composes tracing/usage backends instead of the store (its TraceAsync/CorrelationAsync fetch-isolation catch narrowly excludes a genuine caller cancellation — the method's own cancellationToken actually firing — so it propagates instead of being misreported as an authoritative "not found", while a real backend failure still degrades cleanly to null exactly as before (round 16, #256); the same filter now also covers RecentFlowsAsync and its usage-source fetch loop (TopicsFromUsageAsync), so mesh:query:fleet — the exact call #250 wired a real ambient cancellation token into — propagates a genuine caller cancellation instead of silently degrading to an empty, misleadingly-successful FleetView (round 17, #284)) (Benzene.Mesh.Fleet.Tempo/.Jaeger/.Aws.XRay, Benzene.Mesh.Tracing.Tempo, Benzene.Mesh.Usage.CloudWatch/.ApplicationInsights) — all three trace-source backends now share the same posture on a bounded multi-fetch: per-item fetch isolation (one failing trace or service search is caught, logged, and skipped rather than discarding the rest of an already-successful batch) — the Jaeger/Tempo isolation catch and X-Ray's FetchBatchAsync both distinguish a per-request HttpClient timeout (which throws an OperationCanceledException subclass regardless of the caller's own token) from genuine caller/host cancellation via a token-verified filter (ex is not OperationCanceledException || !cancellationToken.IsCancellationRequested, mirroring MessageHandler.cs's ex.CancellationToken.IsCancellationRequested convention) rather than exception type alone, so one slow backend is isolated exactly like a connection failure while a real ambient cancellation still propagates instead of being swallowed or silently degraded to "not found"/summary-plane rows (round 16, #252); a bounded-concurrency knob (TempoTraceSourceOptions.SearchConcurrency, JaegerTraceSourceOptions.SearchConcurrency) instead of a fully sequential fetch, and a logged warning — never a silent truncation — when a search result hits its configured cap (TempoTraceSourceOptions.SearchLimit, JaegerTraceSourceOptions.SearchLimitPerService, X-Ray's hard-pagination-cap warning); XRayTraceSource.GetCorrelationAsync/GetRecentFlowsAsync de-duplicate fetched trace-summary ids by Id immediately after the fetch (round 17, #274), so a duplicate surfaced across a chunked window's touching boundary (or backend pagination re-surfacing a trace under eventual consistency) can no longer render as two identical TraceViews in a correlation search or silently displace a genuinely different trace from the recent-flows top-N; the pull-based catalog pipeline — Benzene.Mesh.Aggregator + Benzene.Mesh.Contracts + Benzene.Mesh.Artifacts, discovery via Benzene.Mesh.Discovery.Aws/.Azure/.Kubernetes, artifact stores Benzene.Mesh.Aws.S3/.Azure.Blob/.GoogleCloud.Storage, with failure isolation at every fan-out in the pipeline — one failing IMeshDiscoveryProvider (each individually timed and try/caught, MeshDiscoveryRunner.PerProviderTimeout), one failing per-function AWS Lambda tag read, or one throttled previous-snapshot read never costs the rest of the run its results, and Benzene.Mesh.Discovery.Host still refuses to publish outright only when every configured provider failed (a partial-provider failure still publishes, with the failures surfaced); MeshAggregator.RunOnceAsync itself now holds the single-writer gate (every call site serialises through it, not just hosts that remembered their own), stamps every artifact of one run with the same timestamp as a de facto run id, and publishes manifest.json last so a reader never sees it reference an artifact that hasn't landed; a spec/health fetch failure additionally classifies as permission/unreachable/timeout/other (MeshServiceSnapshot.ErrorClass) via each SDK's own status-code shape, read without a compile-time dependency on any of them; KubernetesApiServiceLister paginates (limit+continue) rather than silently truncating at the API server's first page; FileSystemMeshArtifactStore writes are atomic (temp file + rename), so a concurrent read of the same file it's writing never observes a torn value; a topic's request/response/message schema is diffed version-over-version via JsonSchemaComparer into the published catalog's MeshTopicCompatibility verdict (breaking/warning/compatible), correctly seeing through InlineSchema's pre-comparison $ref inlining so a pure oneOf/anyOf/allOf branch reordering reads as compatible rather than a fabricated breaking change (round 11, #152); opt-in production-gated live dispatch (Benzene.Mesh.Dispatch, one declared service, never a shared queue; the real dispatch call resolves its cancellation token via the ambient ICancellationTokenAccessor — the same idiom HttpBenzeneMessageClient uses — so UseTimeout(...) wrapping UseMeshDispatch() genuinely bounds it (round 12, #185); a thrown dispatch exception is audited exactly like every other exit path before the handler returns a ServiceUnavailable result, never a silent raw throw (round 12, #186); an unregistered/arbitrary target service name is rejected by the registry check before the per-target rate-limiter window is ever charged, so it cannot pin a permanent dictionary entry (round 12, #187); the NotImplemented exit path (a service registered against a Source with no matching IMeshServiceDispatcher wired in — the routine "forgot to call AddMeshLambdaDispatcher()" post-deploy misconfiguration) now leaves the same audit record every other exit path does, under its own "no-dispatcher" outcome label, rather than vanishing from the trail (round 16, #255); MeshDispatchRateLimiter.Prune()'s stale-window removal is a compare-and-remove (the conditional TryRemove(KeyValuePair<TKey,TValue>) overload, keyed on both key AND value) rather than an unconditional remove-by-key, so a Prune() decision made from a stale enumeration snapshot can never delete a window a concurrent TryAcquire installed for the same key between that read and the removal executing — closing a hot-path race (Prune runs before every guarded TryAcquire) that could otherwise silently lose a charge and let more than MaxPerMinutePerIdentity/MaxPerMinutePerTarget through for the rest of the minute (round 16, #254)); OIDC login gate for the whole mesh surface (Benzene.Mesh.Auth.Oidc, Authorization Code flow + email allowlist, a Validate()-time fail-fast on a non-https issuer matching MeshAuthGate's, POST /mesh/auth/logout requiring the X-Benzene-Logout header (405 on GET), a per-scope IOidcSessionSink identity handoff, and MeshOidcOptions.Validate() also rejecting a low-entropy SigningKey — fewer than 8 distinct byte values across its length, OR (round 17, #286) an exact tiling of a shorter block whose period is under half the key's total length, closing the gap where e.g. an 8-byte block repeated to fill 32 bytes cleared the distinct-byte floor while still being only 64 bits of real keyspace — since that one key signs both the CSRF state token and the deterministic, stateless session cookie); deploy/Mesh/Benzene.Mesh.Host — the config-driven, dockerized aggregator+UI deployable composing all of the above from mesh.json, gated by its own MeshAuthGate (none/proxy/basic/oidc, proxy mode's trusted-peer check normalizing IPv4-mapped IPv6 addresses so it works on a dual-stack listener (e.g. Kestrel bound to [::], the container default), with a startup-time mode×option satisfiability check — requiredGroups/dispatchRole/allowedEmailDomains/dispatch.enabled each rejected under whichever modes can't satisfy them, plus dispatchRole separately rejected when dispatch.enabled is false (the role check gates an endpoint that doesn't exist without it) — never silently under-enforced, never inert; the dispatchRole check itself is matched via the same path-OR-topic predicate MeshDispatchGuardMiddleware.IsGuarded uses (MeshPathCanonicalizer.IsPathOrTopicMatch), not a literal-path-only comparison, so a future route alias resolving to the dispatch topic under a different path cannot bypass the role requirement (round 17, #287); auth.oidc.requireHttpsMetadata for the non-https-authority case; POST /mesh/auth/logout with IdP end-session redirect; UseMeshDispatchGuard bounds a dispatch request by its actual buffered byte count rather than a trustable-only-in-theory Content-Length header (chunked-transfer-proof), backed by Kestrel's own MaxRequestBodySize as defence-in-depth; MeshArtifactMiddleware/MeshTimeRangeResolver degrade a transient artifact-store failure or an out-of-range relative time query to a clean 503/absent bound respectively, never an unhandled exception on the dashboard's read path — see deploy/Mesh/CONFIG.md; as of round 17, this host's HTTP envelope transport (Benzene.Http's BenzeneMessageHttpMiddleware, which mounts both mesh:query:* and mesh:dispatch here) resolves the OUTER per-request ICancellationTokenAccessor and threads its token through the 3-argument HandleAsync overload into the pipeline's own freshly-created inner scope, so a genuine caller/request cancellation actually reaches the handler — previously it called the 2-argument overload, which hardcodes CancellationToken.None into that inner scope regardless of the outer request's real signal, silently making the #250/#185 cancellation-bounding fixes described above inert for every HTTP-mounted mesh:query:*/mesh:dispatch call on this host, and on any other host built on the same UseBenzeneMessage HTTP-envelope pattern (round 17, #285)) ArtifactStoreMeshReportPublisher/FileSystemMeshArtifactStore.ResolveWithinRoot reject any literal ./.. path segment in a mesh:report name before it ever reaches the store, closing a path-traversal hole where an untrusted report.Name (e.g. "../manifest") could overwrite a sibling top-level artifact like manifest.json (round 14-15, #242); MeshDispatchRateLimiter self-prunes past a size threshold even without the sibling guard middleware calling Prune(); HttpMeshServiceDispatcher caps the target's response the same way the request side has always been capped, truncating with an audit-visible marker rather than buffering an unbounded body. Durable storage for the spec collector — it is a diagnostic surface, not a system of record; the durable output is the aggregator's published catalog artifacts, not collector state. Enforce an upper bound on MeshOidcOptions.SessionDuration — Benzene.Mesh.Auth.Oidc logout is stateless (no server-side session store, only a random Jti carried for a future deny-list that nothing reads yet), so a leaked/stolen session cookie stays valid until its own expiry regardless of the holder logging out; a deliberate, documented tradeoff (a hard cap could break a deployment's genuinely-needed longer "remember me" duration), not an oversight Any port's collector can host this port's services — the benzene:mesh:* wire contract (pinned by the conformance fixtures) is the interop point
Mesh — UI Benzene.Mesh.Ui — serves the canonical mesh-ui.html as an embedded resource (default path /benzene/mesh-ui) (byte-identical to the main repo's vendored copy; the mesh-ui-drift-check.yml workflow fails CI if it drifts from the benzene-ui build), plus mesh-spec-ui.html, the per-service spec view the estate page links to Render its own page — this port ships the shared cross-language UI; the benzene:mesh:query:* contract, not the page, is the interop point for any other UI Point any UI at the collector's query topics
Health checks Benzene.HealthChecks.Core (IHealthCheck.ExecuteAsync(CancellationToken) — the token is required, forwarded by every shipped check into its own I/O, so a per-check timeout genuinely cancels the underlying call instead of merely abandoning it; check/result/builder seams) + Benzene.HealthChecks reserved-topic middleware: UseHealthCheck (healthcheck), UseLivenessCheck/UseReadinessCheck (liveness/readiness, deliberately non-overlapping topics), UseContractsCheck (contracts), with built-in memory, timeout-wrapper and shutdown-readiness checks. A health query whose checks ran answers status ok whether or not the service is healthy, with isHealthy carrying the verdict (spec wire-contracts §5, spec-0.2.0 — no more successful service-unavailable); every HTTP route mapped to a health topic answers 503 when isHealthy is false and 200 otherwise (a warning stays 200), the health report as the body either way (HttpStatusCodeResponseHandler reads the IHealthVerdict payload) — pinned by problem-details-cases.json's httpRules.healthEndpoint; dependency checks Benzene.HealthChecks.Disk/.Http/.Tcp/.DynamoDb/.EntityFramework/.Schema/.Azure.ServiceBus, plus the AWS/Azure client packages' own (SqsHealthCheck, SnsHealthCheck, EventBridgeHealthCheck, …), all relying purely on the processor's uniform timeout rather than an internal one; consumer-side downstream checks (Benzene.Clients.HealthChecks — reachability + contract drift); the ASP.NET Core/grpc.health.v1 bridge (BenzeneHealthCheckBridge in Benzene.Grpc.AspNet). Every shipped check reports a caller-driven cancellation (ambient shutdown, or the processor's own per-check timeout) as a distinct "Cancelled" outcome via ExceptionHandlingHealthCheck, never as an ordinary transient dependency failure — fixed once in HealthCheckError.Classify for the checks that route through it, plus an explicit catch (OperationCanceledException) { throw; } guard ahead of the catch-all in every check that instead builds its own failure result directly (TcpHealthCheck, DynamoDbHealthCheck, HttpPingHealthCheck, the two Benzene.HealthChecks.EntityFramework checks, ClientHealthCheck, CacheHealthCheck), plus a matching per-check guard in the two checks (GrpcHealthCheck, RabbitMqHealthCheck) that also enforce their own connect/round-trip budget on top of the caller's token, so that budget elapsing still correctly reports an ordinary Failed. HttpPingHealthCheck additionally reports Url + the dependency entry (not just an opaque exception) when the endpoint doesn't respond at all (HttpRequestException), the same as it already did for a non-2xx status — the failure mode where an operator running several ping checks most needs to know which endpoint is down. HealthCheckProcessor itself never lets one check's own broken Timeout/IsNonCritical/Type getter crash the whole aggregation — a throwing getter degrades to a Failed result for that one check, every other check's real result is unaffected. CachingHealthCheckProcessor single-flights concurrent callers per cache key (ConcurrentDictionary<string, Lazy<Task<...>>>), so a cold-cache (or just-expired) stampede runs the inner checks exactly once no matter how many callers arrive concurrently, rather than once per caller. BenzeneHealthCheckBridge (Benzene.Grpc.AspNet) duplicates HealthCheckProcessor.RunTimedAsync's non-critical-failure downgrade locally (it deliberately doesn't reference Benzene.HealthChecks), so the two independent aggregation paths agree on a non-critical/non-persistent failing check PerformHealthChecksAsync takes no ambient CancellationToken — a slow check always runs to its own timeout and won't observe a caller disconnect or host shutdown/drain early. Ship a dedicated /health HTTP route — health is a message topic; the HTTP transport maps the report's isHealthy to 200/503 (Benzene.CloudService maps GET /benzene/health; liveness/readiness routes are the app's choice). Wire contract checks into probes — a drifting or struggling downstream must never restart otherwise-healthy pods, so contracts is a diagnostic topic, kept separate from liveness/readiness by design Implement IHealthCheck for any other dependency; see Kubernetes health checks
Spec endpoint & Cloud Service Profile Benzene.Schema.OpenApi — the derived spec from registered handlers/endpoints, served on the benzene:spec topic in three formats (the Contract Document, OpenAPI 3.0, AsyncAPI 3.0) as JSON or YAML — an absent/empty/null/{} request is the contract document, values compare case-insensitively, and an unsupported type or format answers not-implemented; GET /benzene/spec is an ordinary HTTP mapping onto benzene:spec, so it runs the pipeline (and auth) and its type/format query parameters reach the request (contract-document §7). The Contract Document (spec 3a, contract-document.md; passes derivationCases, namingCases, specTopicCases, specHttpCases and every non-projection hash case): schemaDialect: "openapi-3.0"; service/serviceVersion/versionScheme/description from IApplicationInfo, the single source; requests from handlers (no benzene: topic is ever listed; a no-payload handler has no response), events from event declarations (AddResponseEventDeclarations), sends from send declarations (MeshOutboundRegistry via AddSendDeclarations / the cloud service's WithSends), a (topic, version) in both failing the build of the document; both hashes (contractHash, registrationHash) always present, computed by ContractHash (RFC 8785 via JsonCanonicalizer, both now in this package); examples are author-supplied only (none synthesized). Schemas stay Swashbuckle-derived with SupportNonNullableReferenceTypes + UseAllOfToExtendReferenceSchemas + DerivationSchemaFilter: a nullable named type is the {type, nullable, allOf:[$ref]} wrapper, records carry no additionalProperties, required is every always-emitted property, enums are typed and format kept. Component names are the simple type name or a declared [SchemaName] (Benzene.Abstractions.Schema); a collision throws ComponentNameCollisionException naming both types, the topics and the remedy — on the first spec request, or at start-up with AddContractDocumentStartUpCheck() (opt-in for manual composition; on by default under UseBenzeneCloudService, whose WithoutContractDocumentStartUpCheck() spares a Lambda cold start), or at build time in benzene-descriptor. spec-0.4.0 (Phase 3b): the document carries the deployment-side fields — runtime (dotnet), placement, profile — from a registered ContractDocumentDeployment (SetContractDocumentDeployment; the cloud service registers one) and degraded (requests when no handler registry is wired), all outside both hashes; it is also the mesh register body, so derivationCases (now carrying serviceInfo: identity and placement, subset-matched) cover the registration. Notes: the reserved benzene:ping reachability probe is answered by the router itself (ok, no payload, any version; DisablePing() opts out); SchemaBuilder's reflection-based generation (JsonSerializerDataContractResolver) camelCases property names to match the wire, and the spec's own example block — all agree on casing (round 11, #169; previously the schema alone was PascalCase, self-contradicting the others). Two ripples of that fix, closed in the same round: OpenApiValidationSchemaBuilder now camelCases a validation rule's own (reflected, PascalCase) member name before matching it against the now-camelCase schema property keys — left unfixed, every validation facet (Required, MinLength, Pattern, ...) would have silently stopped attaching to any schema at all; JsonOpenApiSchemaBuilder (the JSON-literal schema inferrer behind AddJsonEvent) capitalizes a nested object's inferred component id, since a schema id is conventionally a type name, not a (now camelCase) JSON property name; the envelope endpoint (Benzene.Http's UseBenzeneMessage / BenzeneMessageHttpMiddleware) answers HTTP 200 for every envelope it processed whatever its status, 400 (problem+json) only for a body that is not an envelope, runs the same pipeline (and auth) as the other transports, and restricts reachable topics with BenzeneMessageHttpOptions.TopicFilter (a rejected topic is a not-found envelope) — pinned by problem-details-cases.json's httpRules.envelopeEndpoint; Benzene.Spec.Ui — the topic-centric spec viewer page; every default path sits under /benzene/ (spec-0.2.0, transport-bindings §1): the envelope endpoint /benzene/invoke, the Spec UI /benzene/spec-ui fetching /benzene/spec?type=benzene, the mesh UI /benzene/mesh-ui, MCP /benzene/mcp — each configurable; Benzene.CloudService — UseBenzeneCloudService(...): one call wiring the profile's R1–R5 and R7 (spec-0.5.0; R6/R8 left the profile, ids not reused) at the /benzene/* default paths (invoke envelope; R5's benzene:spec, answered wherever the transport carries a reply, and /benzene/spec; health) and, given a collector, the meshed-service claim's feeds (mesh §7.1: M1 trace feed + collector registration of the served contract document + heartbeats carrying its registrationHash and serviceVersion; M2 the inbound trace join), and the contract-document start-up check by default; it produces an honest wiring-time CloudServiceProfileReport — the profile's Requirements/Missing (stamped on the contract document's profile.missing, from R1–R5/R7 only — no collector needed, so the default wiring is a full Cloud Service) and, apart and never in profile.missing, the meshed-service claim's MeshRequirements (M1 feeds, formerly R6; M2 propagation, formerly R8)/MeshMissing/IsMeshed (which requires the profile); with WithHandlers a reduced document (its events/sends named degraded) registers at wire-up, before any traffic, and the first invocation re-registers the complete one — M2 reads the exact same resolved trace-exporter value Extensions.cs wires UseMeshTrace with (round 11, #167). M2: satisfied when the inbound join is wired and every outbound route forwards the mesh span with UseMeshTraceContext() (sends and published events alike, which share the IBenzeneMessageSender route mechanism — Benzene.ResponseEvents publishes through it), or there are no outbound routes; it cannot see a per-route UseMeshTraceContext() or routes registered after the cloud service is wired, and its note says so. The AwsMesh example forwards only UseW3CTraceContext() (Activity.Current) to keep its X-Ray traces whole, so it reports M2 missing; Benzene.CloudService.Probe — external black-box probe over real HTTP against any running service, .NET or not: R1–R5 and R7 (R5 = /benzene/spec and benzene:spec over the envelope, which also evidences R2's topic count), all observable, so a conformant service is fully Satisfied; M1/M2 reported apart as Inconclusive (not observable from one service), never counted toward --fail-on; a stale R6/R8 in an older service's profile.missing is carried, never interpreted (no .NET consumer reads the ids); Benzene.Descriptor — the benzene-descriptor dotnet tool emitting the contract document {name}.spec.json from a built, non-running service — the .service.json descriptor is retired (--emit descriptor/both fail with a message); --service-version/--version-scheme reach serviceVersion/versionScheme (#171 closed), --cloud/--region reach placement, runtime is dotnet Make Benzene.CloudService load-bearing — it is deliberately sugar-only; no capability lives in it, manual composition of the same Use* calls remains the full-control path, and overrides are reported honestly (relocated paths flagged in the report, never refused) The report's unsatisfied requirements are the exact to-do list; the probe audits from outside what the self-report claims
Codegen & typed clients Benzene.CodeGen.Core/.Client — typed C# client-SDK generation from a service spec (schema closure, canonical JSON + contract hash). Spec 3a (passes parseCases, topicScopeCases, schemaClosureCases and every contract-hash-cases.json case, projections included): the reader rejects an unknown schemaDialect (UnknownSchemaDialectException) and a missing one (spec 1.1, passes missing-schema-dialect-is-rejected) — a legacy (spec-0.2.0) document is read only through the explicit opt-in DeserializeAllowingLegacyShape, with a warning, never by the conformant Deserialize the CLI and the gate use; benzene: entries are ignored and IncludeReservedTopics is gone; an include-list projection (TopicScope.Apply) and a topic-scoped one (TopicScope.ProjectTopic) recompute both hashes, and a generated client's HashCode is the contractHash of its projection (a default client equals the served value); the nullable-reference wrapper is seen through, and an unconstrained {} schema is typed object; the compatibility gate (SchemaCompatibilityComparer, benzene diff) ignores benzene: entries and refuses a legacy baseline with one LegacyBaselineException (re-baseline) rather than false breaking changes; the provider side of the drift check, Benzene.HealthChecks.Schema.SchemaHealthCheck, publishes the served document's contractHash (no private hash, and no codegen dependency in a deployed service); the benzene CLI (Benzene.CodeGen.Cli + .Cli.Core) with spec/build/diff/healthcheck/cloud-service-profile/lambda-test-tool commands over file/HTTP/Lambda/mesh spec sources — diff, healthcheck and cloud-service-profile are all genuine CI gates: each throws (→ non-zero exit) when its report trips the requested --fail-on threshold (diff's default is breaking; healthcheck's and cloud-service-profile's default to failing on an unhealthy/not-satisfied target), never silently exiting 0 regardless of the target's real state; healthcheck tolerates a non-JSON or empty response body (a target not running UseHealthCheck()'s standard shape, or an intermediate proxy/adapter error page) by printing it verbatim and treating it as "not tripped" rather than crashing the whole invocation with a raw JsonReaderException (round 17, #282); build's generated C# always compiles even for a top-level polymorphic (oneOf) request/response schema (typed as the subtypes' shared base when discoverable, else object) and a schema property name that isn't already a valid C# identifier (e.g. order-id → Orderid) — and, for the same reason, a $ref'd schema id that isn't already a valid/Pascal-cased C# identifier (any bring-your-own-schema SuppliedSchemaCatalog id is caller-supplied, arbitrary text) generates a property/parameter type name formatted exactly the way its class declaration is, so the two can never diverge into an uncompilable mismatch (round 15, #240); a discriminator's PropertyName and every mapping.Key — arbitrary caller-supplied strings (the discriminator value, not an identifier), reachable via SuppliedSchemaCatalog or any hand-built EventServiceDocument, not only reflection-derived schemas — are escaped through a small local C#-string-literal escaper before landing in the generated [JsonPolymorphic]/[JsonDerivedType] attributes, so a "/\/control character in a mapping value (e.g. 12" wheel) can no longer produce uncompilable generated C# (round 16, #263); an enum-shaped schema generates a real C# enum (a string enum gets [JsonConverter(typeof(JsonStringEnumConverter))] and its values as member names; an integer enum gets its values as explicit member values), not an empty class with no members (round 11, #166 — the prior empty class sent "field":{} on the wire and a real server rejected it with HTTP 400); topic-scoped client generation (-topics ...) narrows the generated DTOs (and the contract hash) to exactly what SchemaClosure.Reachable finds from the surviving topics, matching AtomicClientSdkBuilder's per-topic narrowing, instead of emitting the whole service catalogue regardless of scope (round 11, #170); diff's SchemaCompatibilityComparer recurses into additionalProperties (a Dictionary<string, T>-shaped schema's value type) the same way it already does for array items, so a breaking change entirely inside a map's value schema is caught rather than passing as "No changes" (round 11, #168); Benzene.CodeGen.Markdown renders a polymorphic property as its shared base type name or a oneOf: {A|B} union listing rather than leaving it blank, and null-checks an array schema's Items before dereferencing it in MapProperty (mirroring GetPropertyTypeName's existing guard for the same shape), rendering the Void[] placeholder instead of throwing NullReferenceException; the same generator always names a type: object-with-additionalProperties-and-no-own-properties field (the shape for a reflected Dictionary<string, T> property) and a genuinely empty object, rather than dropping the property name entirely into a bare, unattributable "{}"/"{}[]" line — where there's a value type to report, it renders the map shape (e.g. scores: {[string]: int}) mirroring CSharpTypeName.GetName's Dictionary<string, T> handling for the C# generator (round 16, #265, minor); Benzene.CodeGen.ApiGateway's ApiGatewayBuilderV1 fails loudly (mirroring ReflectionHttpEndpointFinder's duplicate-route check, including that check's own case-folding of Method — "GET" and "get" on one path collide too, not only identical casing) when two topics collide on the same HTTP method+path, rather than emitting duplicate-key YAML and a corrupted CORS header, and routes every user-controlled value it interpolates into the generated YAML (topic names into summary:, path-derived tags, the configured CORS allow-headers value) through a YAML-safe escaping helper rather than raw string interpolation, so a " in a topic name or a : in a path segment can no longer produce YAML a real parser can't load; and EventServiceDocumentDeserializer builds a fresh schema catalogue per Deserialize() call, so reusing one instance across two documents that share a schema name (e.g. diff's baseline vs. current) never lets the second document's schema silently resolve to the first's (round 11, #46); spec's OpenApiDocumentBuilder fails loudly with a descriptive InvalidOperationException naming both the invalid HTTP method and the offending topic/path when a [HttpEndpoint]'s Method isn't one of OpenAPI's 8 supported operation verbs (GET/PUT/POST/DELETE/OPTIONS/HEAD/PATCH/TRACE, matched case-insensitively), rather than an opaque KeyNotFoundException naming neither (round 15, #241); AddJsonEvent's example-driven schema inference tolerates an ordinary empty example array ("tags": []) with an untyped items placeholder, rather than crashing with InvalidOperationException: Sequence contains no elements (round 15, #242); the same builder now also handles an ordinary JSON decimal number (JTokenType.Float, e.g. a price/percentage/rating — mapped to an OpenAPI number schema) and an ordinary JSON null value (JTokenType.Null — mapped to the same untyped/Nullable = true placeholder convention #242 established for an empty example array), rather than throwing Exception("No map for Float"/"No map for Null") and aborting the whole document (round 16, #264); diff's SchemaCompatibilityComparer/JsonSchemaComparer correctly key a discriminator-mapped inline (non-$ref) oneOf/anyOf member by its actual mapping entry — pairing the n-th inline member with the n-th discriminator-mapping entry not already claimed by a $ref member — instead of a dead fallback comparison that always missed and silently degraded to pure array position, which reported a byte-identical reordering of inline discriminated-union variants as spurious breaking changes (round 15, #239); a baseline document missing events/requests entirely deserializes to empty arrays (matching the sibling transports/tags getters) instead of throwing ArgumentNullException (round 15, #243, minor); Benzene.CodeGen.Build — MSBuild targets compiling a client from a committed {service}.spec.json on every build; Benzene.CodeGen.ApiGateway/.Markdown/.LambdaTestTool; Benzene.Azure.Function.SourceGenerators — AzureFunctionTriggerGenerator emits the [Function]/[…Trigger] boilerplate class per transport from [assembly: BenzeneXTrigger(...)] declarations. Emission runs through one RegisterSourceOutput per transport (restored in WP-C, #38 — a prior round's merge into one shared output both re-emitted every trigger class on any single trigger edit and was the build-crash vector below), while the BENZ0001 name-collision check stays a single global view combined into every transport's output, so a name shared across transports (e.g. a queue trigger and a Kafka trigger both named "dup") is still caught, checked over the full declared set — including a trigger that carries its own diagnostic below (WP-C, #32: previously such a collision could report only that trigger's own diagnostic and silently ship the other side unchecked). Every trigger's Location is an external one (file path + span, no SourceTree — AttributeReading.AttributeLocation), not the tree-bound SyntaxNode.GetLocation() result — TriggerInfo.Location is deliberately excluded from the type's cache-driving equality, which without this let a stale, tree-bound Location survive into a later compilation that no longer contained its tree, crashing the whole incremental build with ArgumentException inside Roslyn's own suppression-checking on an ordinary, unrelated edit (WP-C, #38 — the reproduced build-crash finding; a Location with no tree reference at all has nothing left to go stale). Diagnostics, from one shared DiagnosticDescriptors table: BENZ0001 (error, cross-transport collision, as above; the Function name is deliberately never auto-renamed the way the generated class name is, because it's externally meaningful — bindings, host.json, scale rules, the portal's identity for the function — so silently picking a different one would only move the failure from build time to deployment time); BENZ0002 (error) when a [assembly: BenzeneCosmosDbTrigger(...)] omits the required DocumentType; BENZ0003–BENZ0007 (error, WP-C #39) the same "required field missing" check extended to every other transport with one — ServiceBus (neither QueueName nor TopicName set), EventHub (EventHubName), Kafka (Topic), Queue Storage (QueueName), Blob Storage (Path); BENZ0008 (error, WP-C #40) an explicitly-set empty/whitespace-only Name, across all 9 transports (distinct from the absent case, which correctly defaults); BENZ0009 (warning, WP-C #42) a ServiceBus trigger setting both a queue and a topic — the queue still wins (unchanged precedence), this only surfaces what used to be a silent discard of the topic. All of BENZ0002–BENZ0009 skip emitting the offending trigger except BENZ0009, which warns but still emits. The runtime client layer — Benzene.Clients (IBenzeneMessageClient + batch-send abstractions) with transport clients Benzene.Clients.Http, .InProcess, .Aws.{Lambda,Sqs,Sns,EventBridge,StepFunctions}, .Azure.{ServiceBus,EventHub,EventGrid,QueueStorage}, .GoogleCloud.PubSub Additionally (round 14-15): Benzene.CodeGen.Core/.Client — typed C# client-SDK generation from a service spec (schema closure, canonical JSON + contract hash); the benzene CLI (Benzene.CodeGen.Cli + .Cli.Core) with spec/build/diff/healthcheck/cloud-service-profile/lambda-test-tool commands over file/HTTP/Lambda/mesh spec sources — diff, healthcheck and cloud-service-profile are all genuine CI gates: each throws (→ non-zero exit) when its report trips the requested --fail-on threshold (diff's default is breaking; healthcheck's and cloud-service-profile's default to failing on an unhealthy/not-satisfied target), never silently exiting 0 regardless of the target's real state; healthcheck deliberately treats a response with no isHealthy field at all as healthy by default (don't fail-loud on a shape this tool doesn't recognize) - --strict (round 14-15, #264) flips just that one case, for a CI pipeline that wants a misconfigured target (e.g. a --lambda-name answering with unrelated 200 JSON) caught instead of silently passed; build's generated C# always compiles even for a top-level polymorphic (oneOf) request/response schema (typed as the subtypes' shared base when discoverable, else object) and a schema property name that isn't already a valid C# identifier (e.g. order-id → Orderid); an enum-shaped schema generates a real C# enum (a string enum gets [JsonConverter(typeof(JsonStringEnumConverter))] and its values as member names; an integer enum gets its values as explicit member values), not an empty class with no members (round 11, #166 — the prior empty class sent "field":{} on the wire and a real server rejected it with HTTP 400); topic-scoped client generation (-topics ...) narrows the generated DTOs (and the contract hash) to exactly what SchemaClosure.Reachable finds from the surviving topics, matching AtomicClientSdkBuilder's per-topic narrowing, instead of emitting the whole service catalogue regardless of scope (round 11, #170); diff's SchemaCompatibilityComparer recurses into additionalProperties (a Dictionary<string, T>-shaped schema's value type) the same way it already does for array items, so a breaking change entirely inside a map's value schema is caught rather than passing as "No changes" (round 11, #168); Benzene.CodeGen.Markdown renders a polymorphic property as its shared base type name or a oneOf: {A|B} union listing rather than leaving it blank; Benzene.CodeGen.ApiGateway's ApiGatewayBuilderV1 fails loudly (mirroring ReflectionHttpEndpointFinder's duplicate-route check) when two topics collide on the same HTTP method+path, rather than emitting duplicate-key YAML and a corrupted CORS header; and EventServiceDocumentDeserializer builds a fresh schema catalogue per Deserialize() call, so reusing one instance across two documents that share a schema name (e.g. diff's baseline vs. current) never lets the second document's schema silently resolve to the first's; Benzene.CodeGen.Build — MSBuild targets compiling a client from a committed {service}.spec.json on every build; Benzene.CodeGen.ApiGateway/.Markdown/.LambdaTestTool; Benzene.Azure.Function.SourceGenerators — AzureFunctionTriggerGenerator emits the [Function]/[…Trigger] boilerplate class per transport from [assembly: BenzeneXTrigger(...)] declarations. Emission runs through one RegisterSourceOutput per transport (restored in WP-C, #38 — a prior round's merge into one shared output both re-emitted every trigger class on any single trigger edit and was the build-crash vector below), while the BENZ0001 name-collision check stays a single global view combined into every transport's output, so a name shared across transports (e.g. a queue trigger and a Kafka trigger both named "dup") is still caught, checked over the full declared set — including a trigger that carries its own diagnostic below (WP-C, #32: previously such a collision could report only that trigger's own diagnostic and silently ship the other side unchecked). Every trigger's Location is an external one (file path + span, no SourceTree — AttributeReading.AttributeLocation), not the tree-bound SyntaxNode.GetLocation() result — TriggerInfo.Location is deliberately excluded from the type's cache-driving equality, which without this let a stale, tree-bound Location survive into a later compilation that no longer contained its tree, crashing the whole incremental build with ArgumentException inside Roslyn's own suppression-checking on an ordinary, unrelated edit (WP-C, #38 — the reproduced build-crash finding; a Location with no tree reference at all has nothing left to go stale). Diagnostics, from one shared DiagnosticDescriptors table: BENZ0001 (error, cross-transport collision, as above; the Function name is deliberately never auto-renamed the way the generated class name is, because it's externally meaningful — bindings, host.json, scale rules, the portal's identity for the function — so silently picking a different one would only move the failure from build time to deployment time); BENZ0002 (error) when a [assembly: BenzeneCosmosDbTrigger(...)] omits the required DocumentType; BENZ0003–BENZ0007 (error, WP-C #39) the same "required field missing" check extended to every other transport with one — ServiceBus (neither QueueName nor TopicName set), EventHub (EventHubName), Kafka (Topic), Queue Storage (QueueName), Blob Storage (Path); BENZ0008 (error, WP-C #40) an explicitly-set empty/whitespace-only Name, across all 9 transports (distinct from the absent case, which correctly defaults); BENZ0009 (warning, WP-C #42) a ServiceBus trigger setting both a queue and a topic — the queue still wins (unchanged precedence), this only surfaces what used to be a silent discard of the topic; BENZ0010 (error, round 16-17 #259) a Cosmos DB trigger missing DatabaseName/ContainerName; BENZ0011 (error, round 14-15 #233, renumbered from BENZ0010 on merge — round 16-17's independent Cosmos DB fix claimed BENZ0010 first) a ServiceBus trigger setting a topic without a subscription, or vice versa, with no queue set — either half alone previously passed every check and silently generated a broken binding ([ServiceBusTrigger("audit", "")], syntactically valid but broken at deployment). All of BENZ0002–BENZ0011 skip emitting the offending trigger except BENZ0009, which warns but still emits. The runtime client layer — Benzene.Clients (IBenzeneMessageClient + batch-send abstractions) with transport clients Benzene.Clients.Http, .InProcess, .Aws.{Lambda,Sqs,Sns,EventBridge,StepFunctions}, .Azure.{ServiceBus,EventHub,EventGrid,QueueStorage}, .GoogleCloud.PubSub Ship Benzene.CodeGen.Terraform — deliberately excluded from the release (IsPackable=false) until its grow-or-split lifecycle call is made (it does escape "/\ in every value it interpolates into a generated .tf string literal — TerraformEventBridgeRuleBuilder/.LambdaBuilder/.LambdaEventBusPermissionsBuilder all route through one NameFormatter.EscapeHclString helper, round 15, #244 — but remains otherwise unhardened, unpackaged experimental code). Auto-rename a colliding Function name to make the build pass — rejected: it's the user's name to fix, not the generator's to guess Additionally (round 14-15): Ship Benzene.CodeGen.Terraform — deliberately excluded from the release (IsPackable=false) until its grow-or-split lifecycle call is made. Auto-rename a colliding Function name to make the build pass — rejected: it's the user's name to fix, not the generator's to guess Give the offending trigger a distinct name / its required field(s) / a non-empty Name per the diagnostic; the diagnostic's location points at the offending [assembly: ...] attribute's file/line/column
AWS Lambda-clients Benzene.Clients.Aws.Lambda — two invocation shapes, both classifying the real outcome instead of assuming success: AwsLambdaBenzeneMessageClient (IBenzeneMessageClient) chooses InvocationType from TResponse (Void → Event/fire-and-forget, else → RequestResponse/awaited), and treats both a RequestResponse invoke's InvokeResponse.FunctionError (AWS returns HTTP 200 even when the target function threw) and an Event invoke's non-2xx InvokeResponse.StatusCode (thrown by the lower-level AwsLambdaClient as AwsLambdaFunctionErrorException/AwsLambdaEventInvokeFailedException respectively) as a caught failure (ServiceUnavailable), never a false Accepted; the lower-level UseAwsLambda<T>()/LambdaContextConverter<T> pipeline is fire-and-forget-only, sends InvocationType.Event (not the SDK's RequestResponse default), and maps a 2xx InvokeResponse.StatusCode to Accepted, anything else to a failure result. Plus AwsLambdaHealthCheck (reachability/active modes) — no internal timeout guard, relies purely on the processor's uniform timeout wrap, same shape as SqsHealthCheck/StepFunctionsHealthCheck. Every outbound AWS client in this family — this package's AwsLambdaClient/AwsLambdaBenzeneMessageClient/AwsLambdaClientMiddleware/AwsLambdaHealthCheck, plus Benzene.Clients.Aws.{Sqs,Sns,EventBridge}'s client middlewares and batch clients, Benzene.Clients.Aws.StepFunctions, and the standalone Benzene.Aws.Sqs's SqsMessageClient — now resolves the ambient ICancellationTokenAccessor (an optional constructor overload mirroring HttpClientMiddleware's) and threads it into the underlying AWS SDK call, so UseTimeout(...)/graceful-drain cancellation around an outbound AWS send genuinely aborts it instead of being a silent no-op (round 16, #261 — every AWSSDK.* method already had a CancellationToken overload; nothing here was calling it). A response payload through the UseAwsLambda<T>() pipeline shape — it is fire-and-forget by design; use AwsLambdaBenzeneMessageClient for a request/response call See AWS Lambda clients for both shapes' exact semantics
AWS Step Functions client Benzene.Clients.Aws.StepFunctions — StepFunctionsClient.StartExecutionAsync(message, executionName?) makes a Step Functions start genuinely idempotent: on ExecutionAlreadyExistsException (an execution-name collision), it calls DescribeExecution and compares the existing execution's Input to this call's serialized input as an exact string — a match is a true idempotent duplicate (Accepted); a mismatch is a Conflict failure result (the caller's payload was not started, left untouched, and the caller is told so rather than getting a false-positive Accepted). SanitizeExecutionName keeps the mapping from idempotency token to execution name collision-resistant: whenever character replacement or the 80-character cap would otherwise make the sanitized name diverge from the original, the result is the sanitized name truncated to 71 characters plus an 8-hex-character SHA-256(original name) suffix, so distinct originals that happen to sanitize or truncate alike still land on distinct execution names — required for the DescribeExecution compare above to mean anything. StepFunctionsClient/StepFunctionsClientFactory now resolve the ambient ICancellationTokenAccessor and thread it into both StartExecutionAsync and the DescribeExecution verification call (round 16, #261), so a stalled SDK call is genuinely aborted near a wrapping UseTimeout(...)'s deadline; because the client's own catch-all converts any exception (including a resulting cancellation) into a ServiceUnavailable result rather than rethrowing, the observable effect is the call completing promptly instead of hanging, not a propagated TimeoutException. Reconstruct or resume the workflow itself — this client only starts an execution; see the Sagas / workflows row Callers relying on the idempotency name must serialize input deterministically (same logical input → the same serialized string); requires states:DescribeExecution alongside states:StartExecution — see AWS IAM permissions
Caching Benzene.Cache.Core — provider-agnostic cache-aside/write-through abstractions + base classes your handler code calls directly, plus a cache health check; Benzene.Cache.Redis — the Redis provider. Every member of ICacheService/ICacheEntry<T>/ICacheWriteActions<T>/ICacheInvalidateActions takes a CancellationToken (optional, default), threaded into the Redis calls via Task.WaitAsync (StackExchange.Redis has no native per-call cancellation) and into RedisCacheService's own connect, which previously had no deadline of any kind; LazyLoadAsync/WriteThroughAsync also take an optional per-call expireIn, flowed through to the underlying SetValueAsync. Negative caching is supported: an explicit SetValueAsync(default) is a real, repeatable cache hit on the next LazyLoadAsync (presence — a non-null stored value, never string emptiness — decides a hit, so this holds even for a custom ISerializer that encodes null/default as "" rather than the stock serializer's "null") rather than a permanent miss that re-runs the database read every time. A write-through cache-sync failure (exception, or a provider honestly returning false) after the database write already committed — including one thrown by the caller-supplied cache-value/cache-action mapping delegates on WriteThroughAsync, not just the cache I/O itself — is logged and swallowed, never surfacing as that operation's own failure — the database write is the source of truth. RedisCacheService accepts a constructor-injected ISerializer (falling back to a shared default, not a fresh instance per cache entry) and guards a RedisSetup/connect call arriving after DisposeAsync with ObjectDisposedException rather than silently leaking a new connection. RedisCacheService also implements IDisposable (bridging synchronously to its own DisposeAsync with a bounded 5-second wait, round 16 #262) alongside IAsyncDisposable — the documented "register your subclass so its container disposes it" pattern used to throw InvalidOperationException on the very first message whenever the container's own scope teardown was purely synchronous (the only kind Benzene.Abstractions.DI.IServiceResolver exposes), including through MicrosoftServiceResolverFactory.Dispose() — the only disposal path Benzene.Aws.Lambda.Core has. Separately, both MicrosoftServiceResolverAdapter.Dispose() and MicrosoftServiceResolverFactory.Dispose() (Benzene.Microsoft.Dependencies) now bridge to DisposeAsync() with an unbounded wait whenever the wrapped scope/provider supports it (round 16 #266) — the systemic fix for the same defect class against any user-registered IAsyncDisposable-only scoped/transient/singleton service, not just Redis's. RedisMultiKeyActions issues its per-key writes concurrently (every key is always attempted, independent of another key's failure) and its multi-key invalidate as a single atomic DEL. RedisCacheService's prefix-based invalidation (CreatePrefixActions) rejects a null/empty/whitespace prefix with ArgumentException before building the wildcard pattern, and the pattern-based invalidator refuses to execute a bare/effectively-universal glob (e.g. "*") even if one is somehow constructed — both close off a one-bad-string-away full-keyspace wipe Additionally (round 14-15): Benzene.Cache.Core — provider-agnostic cache-aside/write-through abstractions + base classes your handler code calls directly, plus a cache health check; Benzene.Cache.Redis — the Redis provider. Every member of ICacheService/ICacheEntry<T>/ICacheWriteActions<T>/ICacheInvalidateActions takes a CancellationToken (optional, default), threaded into the Redis calls via Task.WaitAsync (StackExchange.Redis has no native per-call cancellation) and into RedisCacheService's own connect, which previously had no deadline of any kind; LazyLoadAsync/WriteThroughAsync also take an optional per-call expireIn, flowed through to the underlying SetValueAsync. Negative caching is supported: an explicit SetValueAsync(default) is a real, repeatable cache hit on the next LazyLoadAsync (presence, not value-nullness, decides a hit) rather than a permanent miss that re-runs the database read every time — presence is cacheValue is not null exactly, null alone is the miss marker, so a stored empty string (a legitimate serialized representation for some ISerializers) round-trips as a real hit rather than being mistaken for a miss (#201; RedisCacheEntry<T>'s own read-error path was audited to return null, never "", under the same rule). A write-through cache-sync failure (exception, or a provider honestly returning false) after the database write already committed is logged and swallowed, never surfacing as that operation's own failure — the database write is the source of truth, and the same protection now covers the three-arg WriteThroughAsync overload's getCacheAction/getCacheValue mapping delegates themselves (#199), not only the Set/Invalidate step they feed. RedisCacheService accepts a constructor-injected ISerializer (falling back to a shared default, not a fresh instance per cache entry) and guards a RedisSetup/connect call arriving after DisposeAsync with ObjectDisposedException rather than silently leaking a new connection. RedisMultiKeyActions issues its per-key writes concurrently (every key is always attempted, independent of another key's failure) and its multi-key invalidate as a single atomic DEL. CreatePrefixActions throws ArgumentException on a null/empty/whitespace prefix rather than silently invalidating the whole database ("*") — CreateWildcardActions("*") is the explicit, deliberate route for that (#198) Cache middleware or HTTP response caching — what and when to cache is a handler-level decision; no pipeline knob decides it for you. Automatic negative caching of a database miss (LazyLoadAsync still only writes a successful read's payload back; a caller decides for itself whether a null/not-found result is worth caching and calls SetValueAsync(default, ...) itself). Cancel the caller-supplied modifyDatabaseFunc/databaseReadFunc delegates passed to WriteThroughAsync/WriteThroughInvalidateAsync/LazyLoadAsync — the CancellationToken parameter bounds only the cache's own read/write/invalidate I/O, never these zero-arg delegates (a deliberate scope boundary, not an oversight: retrofitting a Func<CancellationToken, Task<TResult>> shape onto them is a second, independently-justified breaking change); a caller needing its own database call cancelled observes the ambient token itself, same as calling that database directly outside the cache Extend the Benzene.Cache.Core base classes for another provider
Validation Three adapters sharing one short-circuit failure contract: Benzene.FluentValidation (runs the app's own DI-registered IValidator<T> before the handler), Benzene.DataAnnotations (BCL attributes, deliberately minimal), Benzene.JsonSchema (raw body vs. a JSON Schema generated from the handler's request type, or supplied, resolved against the version the request actually declares on a multi-version topic — see GetVersionedTopic in Message Handlers); seams in Benzene.Abstractions.Validation. All three now honour the same failure-status override: each resolves an optional IValidationStatusMapper and, when one is registered (only Benzene.FluentValidation ships an implementation, DefaultValidationStatusMapper) and the resolved handler carries [ValidationStatus], uses its status instead of the default ValidationError — register the mapper once (or your own IValidationStatusMapper) and every adapter honours it identically; absent a registered mapper, each keeps its pre-existing default unchanged. All three emit errors[].field as an RFC 6901 JSON Pointer into the request's JSON form (wire-contracts §1.3): FluentValidation and DataAnnotations convert their .NET property paths (Items[0].Sku → /items/0/sku) through FieldPointer, named by the service's own JSON serializer (naming policy and [JsonPropertyName] renames applied); JsonSchema reports the evaluator's instance location; a rule naming no single location (whole-request, cross-field) carries no field; over gRPC the pointer becomes the dotted FieldViolation.field (items[0].sku) and a Benzene gRPC client reads it back as the pointer Define a validation DSL of its own — you write ordinary FluentValidation/attribute/schema rules; Benzene only wires them into the pipeline and maps failures to a validation-status result. The three adapters share the short-circuit contract, not the coverage: DefaultJsonSchemaProvider's generated schema is type-shape-only — it does not understand System.ComponentModel.DataAnnotations attributes ([Required]/[Range]/[MinLength], ...), so a DTO validated meaningfully under Benzene.DataAnnotations/Benzene.FluentValidation can pass Benzene.JsonSchema's default provider with no warning; see the package's README for the gap and how to close it (a supplied schema, or Json.Schema.Generation's own attributes). Any other library via a small middleware honouring the same short-circuit contract
Versioning Both spec mechanisms. Handler-version dispatch: (topic, version) handler registration selected by IVersionSelector — the default VersionSelector (Benzene.Core.MessageHandlers) applies spec core-concepts §2: the exact version, else the handler the version's payload can be cast to (via IPayloadVersionCasts, registered by AddPayloadVersioning; the topic's default wins a tie, otherwise the ordinal-greatest version — a documented deterministic tie-break), else not-found naming topic + version; no "highest available" fallback (kept only as the opt-in HighestAvailableVersionSelector). A message with no version goes to the topic's default handler — the unversioned one, else the one declared with [Message(topic, version, IsDefault = true)] / AddMessageHandler(..., isDefault: true) — else not-found; the version-selection start-up check fails on two declared defaults and warns for a topic with only versioned handlers and no default. The version is read from one header, benzene-version (no version/x-version fallback list), renamed in one place via IBenzeneWireNames.Version / AddMessageVersionHeaderName(...) and written under the same name outbound (WithVersion, MCP); HTTP's /v{version} route segment supplies it as if the header were sent. It is read by the transport-neutral IMessageVersionGetter and joined into the topic once, inside IMessageGetter<TContext>.GetTopic itself (cached per message) — every reader of the topic (the router, UseMeshTrace, health checks, Benzene.CloudService dispatch, ...) sees the same version-resolved ITopic, not only the router. Two concrete implementations carry this join, not one: the open-generic MessageGetter<TContext> facade (Benzene.Core.MessageHandlers) and BenzeneMessageGetter (Benzene.Core.MessageHandlers.BenzeneMessage) — the closed-type registration that wins DI resolution for the BenzeneMessage transport nearly every test in this repo uses, and so needed the identical fix independently. Payload casting: Benzene.Core.Versioning — up- and downcasting with auto-composed multi-step chains (a registered shortcut caster is preferred), serializer-agnostic (JSON/XML/MessagePack/Avro); CasterFuncBuilder supports self-referential and mutually-recursive versioned DTO shapes (a Node.Child: Node tree, two types referencing each other) via a build-time indirection cell installed before recursing into the expression builder and filled in once the recursive caster finishes compiling, rather than recursing unbounded — previously an uncatchable StackOverflowException at Upcast/registration time on the most ordinary tree/graph-shaped payload (round 15, #226); Benzene.Grpc.Versioning adds the request side over gRPC Response-side downcasting over gRPC (see the gRPC row) Swap in your own IVersionSelector for a different selection policy (e.g. HighestAvailableVersionSelector, the pre-spec-0.2 behaviour)
Serialization beyond JSON Benzene.Xml, Benzene.MessagePack, Benzene.Avro — negotiable IMediaFormats alongside JSON via request/response content negotiation; Benzene.NewtonsoftJson — a drop-in Json.NET ISerializer for apps needing Json.NET semantics. Untrusted-input hardening: Benzene.Avro bounds both allocation (BoundedBinaryDecoder rejects a hostile length prefix before allocating) and recursion depth (AvroOptions.MaxDepth, default 500, on both serialize and deserialize — a self-referencing/deeply-nested schema or object graph throws a catchable AvroPayloadTooDeepException instead of an uncatchable CLR stack overflow); Benzene.Avro also fully supports the Avro map type via an explicit/registered schema (values converted recursively, so a map of records/arrays/maps round-trips, not just primitives; a non-string-keyed CLR dictionary target throws NotSupportedException rather than coercing the key, per Avro's always-string-keyed map spec) and resolves a union's branch by the value's/datum's actual runtime type on both serialize and deserialize, so a union with 3+ non-null branches (e.g. ["null","string","long","boolean"]) is no longer silently miscoded through the first-declared branch — the common 2-branch ["null", X] shape is unaffected; Benzene.Xml's reader prohibits DTDs (entity-expansion DoS) and accepts a UTF-8-BOM-prefixed body like every other transport does; it also round-trips a null payload the same way MessagePack/Avro do — Serialize(type, null) → "", and Deserialize of null or "" → null rather than throwing (malformed non-empty XML still throws). The default System.Text.Json serializer and Benzene.NewtonsoftJson now agree on the wire form for NaN/Infinity/-Infinity doubles (both emit the quoted strings "NaN" etc.; STJ used to throw) Additionally (round 14-15): Benzene.Xml, Benzene.MessagePack, Benzene.Avro — negotiable IMediaFormats alongside JSON via request/response content negotiation; Benzene.NewtonsoftJson — a drop-in Json.NET ISerializer for apps needing Json.NET semantics. Untrusted-input hardening: Benzene.Avro bounds both allocation (BoundedBinaryDecoder rejects a hostile length prefix before allocating) and recursion depth (AvroOptions.MaxDepth, default 500, on both serialize and deserialize — a self-referencing/deeply-nested schema or object graph throws a catchable AvroPayloadTooDeepException instead of an uncatchable CLR stack overflow); Benzene.Xml's reader prohibits DTDs (entity-expansion DoS), accepts a UTF-8-BOM-prefixed body like every other transport does, and bounds deserialize recursion depth the same way (XmlOptions.MaxDepth, default 32, via the DepthGuardedXmlReader wrapper — a self-referencing/deeply-nested request DTO throws a catchable BenzeneException instead of an uncatchable CLR stack overflow; #260). The default System.Text.Json serializer and Benzene.NewtonsoftJson now agree on the wire form for NaN/Infinity/-Infinity doubles (both emit the quoted strings "NaN" etc.; STJ used to throw) Carry raw binary bodies — every Benzene transport body is a string today, so MessagePack/Avro Base64-armor their binary wire form (a stated boundary of the current wire contract, not an accident). Benzene.Avro has no schema-evolution support — the writer and reader use the same resolved schema (no schema exchanged on the wire), so reader and writer must share the exact same field shape; a field added/removed/reordered between producer and consumer versions is not a supported change (a same-type field-count/order mismatch is now detected and reported as AvroSchemaMismatchException rather than silently misread or an opaque exception, but this is detection, not resolution) Implement ISerializer/IMediaFormat for any other format
Event sourcing Benzene.EventSourcing — a deliberately tiny IEventStore seam: AppendAsync(streamId, expectedVersion, events) (optimistic concurrency, throws EventStoreConcurrencyException — now with an optional inner-exception constructor preserving the underlying store failure — if the stream moved; a negative expectedVersion is rejected with ArgumentOutOfRangeException before either store touches storage) and ReadAsync(streamId, fromVersion) (ordered events), serialization-agnostic (EventEnvelope/StoredEvent carry an opaque Payload + EventType discriminator, no JSON opinion baked in); InMemoryEventStore for a single process/tests — appends are atomic across a batch (a mid-batch failure leaves the stream and its dictionary entry untouched, never a partial write or a leaked empty-stream registration for a rejected unknown stream id), observe CancellationToken, and enforce the SAME effective per-append ceiling as the DynamoDB store — 100 events for a brand-new stream (expectedVersion == 0), but only 99 when appending onto an EXISTING stream (expectedVersion > 0), mirroring the DynamoDB store's real transact-item constraint below even though this store has no physical condition-check item of its own (#271 contract parity); Benzene.EventSourcing.DynamoDb — the distributed store for a fleet, its constructor fail-fast validating the table/key configuration (non-empty table and key attribute names, distinct partition/sort keys, neither colliding with the reserved eventType/payload/timestamp attributes): one item per event keyed (streamId, version), AppendAsync is a single TransactWriteItems — carrying a deterministic ClientRequestToken derived from the append's own content, so a retried request lands as the same idempotent attempt rather than a fresh conflicting one — with each Put conditional on that slot not existing (attribute_not_exists), plus (when expectedVersion > 0) a ConditionCheck that the stream is genuinely at expectedVersion (attribute_exists), not merely that the slots above it are free — so racing writers on the same expected version can't both win, and an expectedVersion ahead of the real head can no longer silently gap the stream; because that ConditionCheck item itself counts against AWS's 100-item TransactWriteItems limit, the effective per-call cap on event count is 100 for a brand-new stream but only 99 when expectedVersion > 0 — exceeding it throws a friendly pre-flight ArgumentException (no SDK call made) rather than letting AWS's own validation error surface (#271); a TransactionCanceledException is inspected item-by-item and translated to EventStoreConcurrencyException only for a genuine ConditionalCheckFailed/TransactionConflict reason, with the post-conflict "actual version" diagnostic read run under its own CancellationToken.None and degrading to ActualVersion = -1 on its own failure rather than ever replacing the real conflict; an empty append still runs the same concurrency check as a non-empty one (via a direct consistent read, matching InMemoryEventStore); a read event with a missing or non-string eventType/payload attribute throws rather than silently defaulting to an empty string Additionally (round 14-15): Benzene.EventSourcing — a deliberately tiny IEventStore seam: AppendAsync(streamId, expectedVersion, events) (optimistic concurrency, throws EventStoreConcurrencyException — now with an optional inner-exception constructor preserving the underlying store failure — if the stream moved; a negative expectedVersion is rejected with ArgumentOutOfRangeException before either store touches storage) and ReadAsync(streamId, fromVersion) (ordered events), serialization-agnostic (EventEnvelope/StoredEvent carry an opaque Payload + EventType discriminator, no JSON opinion baked in); InMemoryEventStore for a single process/tests — appends are atomic across a batch (a mid-batch failure leaves the stream and its dictionary entry untouched, never a partial write or a leaked empty-stream registration for a rejected unknown stream id), observe CancellationToken, and enforce the same 100-event-per-append ceiling as the DynamoDB store; Benzene.EventSourcing.DynamoDb — the distributed store for a fleet, its constructor fail-fast validating the table/key configuration (non-empty table and key attribute names, distinct partition/sort keys, neither colliding with the reserved eventType/payload/timestamp attributes): one item per event keyed (streamId, version), AppendAsync is a single TransactWriteItems — carrying a deterministic ClientRequestToken derived from the append's own content, so a retried request lands as the same idempotent attempt rather than a fresh conflicting one — with each Put conditional on that slot not existing (attribute_not_exists), plus (when expectedVersion > 0) a ConditionCheck that the stream is genuinely at expectedVersion (attribute_exists), not merely that the slots above it are free — so racing writers on the same expected version can't both win, and an expectedVersion ahead of the real head can no longer silently gap the stream; a TransactionCanceledException is inspected item-by-item and translated to EventStoreConcurrencyException only for a genuine ConditionalCheckFailed/TransactionConflict reason, with the post-conflict "actual version" diagnostic read run under its own CancellationToken.None and degrading to ActualVersion = -1 on its own failure rather than ever replacing the real conflict; an empty append still runs the same concurrency check as a non-empty one (via a direct consistent read, matching InMemoryEventStore); a read event with a missing or non-string eventType/payload attribute throws rather than silently defaulting to an empty string An aggregate framework, rehydration, snapshots, replay, or projections — the store is only the append-only log; folding events into state is a pure app-level function by design, kept minimal on purpose. Provision or create the DynamoDB table — the consumer registers IAmazonDynamoDB and provisions the composite-key table itself Rehydrate by folding ReadAsync in your command handler, then AppendAsync at the version you read; build CQRS projections by consuming the DynamoDB table's own stream via Benzene.Aws.Lambda.DynamoDb ([Message("table:INSERT")]); evolve event shapes with AddPayloadVersioning (upcast on read) — events are immutable, never rewritten
Map-reduce (scatter-gather) Benzene.MapReduce — IBenzeneMessageSender.ScatterGatherAsync<TShard,TPartial,TAccum>(topic, shards, seed, reduce, options?): the map is one SendAsync per shard run concurrently through BoundedFanOut.WhenAllAsync (bounded by MaxDegreeOfParallelism, results folded in source order; each shard resolves through the routing table, so on AWS it's a Lambda-to-Lambda invoke), the reduce is an app-owned fold over the successful partial responses; PartialFailureMode.ThrowOnAnyFailure (default — throws ScatterGatherPartialFailureException so an incomplete total is never mistaken for complete) or .BestEffort (reduces over successes, reports FailedShards/IsComplete). Each failed shard carries its own failure reason — FailedShards is IReadOnlyList<FailedShard<TShard>> (shard + the Exception? its worker call threw, null for an unsuccessful-result failure), and a thrown ScatterGatherPartialFailureException aggregates every reason onto InnerException (an AggregateException) plus a per-shard Failures list, so which shard failed and why is diagnosable from the exception alone. BoundedFanOut.WhenAllAsync itself now accepts an optional CancellationToken (#230 fix) that a queued shard behind MaxDegreeOfParallelism observes while waiting for a free slot — every batch trigger built on it (AzureFunctionBatchApplicationBase, MiddlewareMultiApplication) propagates its own ambient token through; ScatterGatherAsync's own public signature has no CancellationToken parameter yet, so it currently passes default (no shard observes cancellation via this path today) Ship a dedicated scatter-gather engine or framework — this is the thin, supported composition of parts Benzene already has (SendAsync + BoundedFanOut), kept to one method, options, and a result on purpose. Built-in hierarchical fan-out for very large shard counts For very large fan-outs, shard hierarchically yourself — a coordinator scatters to partition workers that each scatter and reduce locally, the same ScatterGatherAsync call at each level
Rate limiting Benzene.RateLimiting — RateLimitingMiddleware<TContext> built directly on System.Threading.RateLimiting's abstract RateLimiter (BYO fixed/sliding window, token bucket, concurrency, or custom limiter); never queues (QueueLimit = 0 on every built-in) so a rejection is immediate, not memory pressure; convenience entry points UseFixedWindowRateLimiting, UseTokenBucketRateLimiting, UsePayloadSizeRateLimiting (a bytes-per-second token bucket costing each message its UTF-8 body size, with a cheap Content-Length-based pre-check — a rate/CPU saving, not a memory bound: see Rate Limiting for why); per-caller partitioning via PartitionedRateLimitingMiddleware<TContext>/UsePartitionedRateLimiting over a BYO PartitionedRateLimiter<TContext>, so one abusive caller can't exhaust the shared budget for everyone else (the key is caller-supplied and therefore spoofable, but strictly better than the single shared limiter every other entry point still uses by default); a rejected message short-circuits with TooManyRequests (→ HTTP 429), carrying the limiter's retry-after metadata in both the error message and a Retry-After response header when the limiter supplies it, and logging a warning when an ILogger is registered, with a source-accurate rejection message distinguishing a disposed limiter from a disposed dependency the permit-cost delegate itself relied on (#202) — both still fail closed; the three convenience limiters are captured directly in their own middleware's closure rather than resolved from DI, so any number of them can be stacked on one pipeline, or used independently across sibling pipelines sharing one IBenzeneServiceContainer, with no risk of one silently shadowing another (#200) — disposal for one of them is the middleware's own to do (RateLimitingMiddleware<TContext>.DisposeAsync), not automatic; a caller-supplied/BYO limiter's disposal is always the caller's, untouched by the middleware regardless Authoritative, fleet-wide rate limiting — every limiter instance is per service instance: a fleet of N instances admits up to N× the configured rate, and serverless scale-out multiplies it further. This package protects one instance from being a free denial-of-service/cost-amplification vector; it is not traffic shaping. UsePayloadSizeRateLimiting is a rate bound on oversized payloads, not a hard memory cap — ASP.NET Core hosts buffer the request body before this middleware ever runs Put authoritative rate limiting at the gateway (API Gateway, APIM, ingress) in front of every instance; put a host-level body-size cap (Kestrel's MaxRequestBodySize, a gateway limit) in front of an endpoint where payload size is a genuine memory-exhaustion concern; see Rate Limiting
Conformance test/Benzene.Conformance.Test runs the vendored language-neutral fixtures in test/conformance-fixtures/ (a SPEC_VERSION-stamped snapshot of the main repo's docs/specification/conformance/): envelope (including the spec-0.2.0 version-selection cases against conformance:versioned), HTTP + gRPC status mappings, status vocabulary, problem details (including httpRules.envelopeEndpoint and httpRules.healthEndpoint over a real AspNet host), contract document/hash, and the mesh trace/issue/collector/service-version/version-order shapes (mesh-descriptor-cases.json and its runner were deleted with the descriptor in spec-0.4.0) (transport-metadata-cases.json runs in test/Benzene.Core.Test/Conformance/: every binding's topic key, the version key, the EventBridge benzene-headers key, and every metadataCases/overrideCases entry — headersExclude and the case-insensitive topic key included — against every metadata binding: SQS (Lambda and self-hosted), SNS, Service Bus (Function and self-hosted), Event Hubs (Function and self-hosted), Pub/Sub, RabbitMQ and Kafka (self-hosted, Lambda, Function), plus the topicSources fallbacks: RabbitMQ's routing key and Kafka's own topic route when the topic key is absent, and the key wins when present). The requires: validation case runs through FluentValidation (ProblemDetailsConformanceTest). The snapshot is pinned to spec-1.2.0 (SPEC_VERSION), every contract-document/hash/collector/service-version/transport-metadata case passing; the conformance-drift-check.yml workflow re-verifies the vendored snapshot against the canonical repo Edit a fixture to make this port pass — the fixture is the neutral truth A mismatch is a bug here or a deliberate spec change there, never a fixture patch

Why "we don't do that" is a feature

Every deliberate "no" above buys you something: you keep the full power of the tool you chose, you're never blocked by a leaky abstraction, and the surface you do depend on stays small and stable. When you need something Benzene doesn't ship, the extension model (custom middleware, custom pipes, the getter/setter mapper pattern) is a supported, documented path — see Middleware and Common Middleware.

If you find yourself wishing Benzene abstracted a transport or a database away, that's usually the signal to reach for the SDK directly inside your handler — which is exactly where Benzene's design wants that code to live.

Edit this page on GitHub