Benzene

Event-Driven Choreography

Status: DRAFT v0.1 — part of the Patterns "composing a system" set.

The two-tier architecture drives processes with orchestrators: a central service issues commands to core services and runs a saga to keep the multi-service write atomic. Choreography is the other way to move a process across a fleet: no central conductor — each service reacts to events and emits its own, and the process emerges from the chain of reactions.

Both are first-class on Benzene, and a real system uses both. This document is about when to choreograph, and exactly how you build it with Benzene's events, transports, and mesh.

Event-driven choreography: a signup service emits tenant:created, and three independent reactions - welcome email, cache warm, and trial clock - each subscribe and react on their own, with no central conductor.

Orchestration vs. choreography

Orchestration (orchestrators.md) Choreography (this doc)
Control Central — one service directs the process Distributed — each service decides its own reaction
Coupling Orchestrator knows every step Emitter knows nothing about who reacts
Atomicity A saga: all-or-nothing across services None built-in — each reaction succeeds/retries on its own
Adding a step Edit the orchestrator Add a new consumer; no existing service changes
Best for An invariant that must hold across services (a signup that must not half-apply) Fan-out reactions that are independently retryable (on signup: send a welcome email, warm a cache, start a trial clock)

The rule of thumb: orchestrate what must be atomic; choreograph what must merely happen. A process usually has both — a small atomic core (orchestrated as a saga) that emits an event, and a spray of independent reactions (choreographed). Do not use choreography to hold a cross-service invariant; there is no central rollback, and reconstructing one out of scattered compensations is exactly the complexity the saga exists to remove.

How you build it with Benzene

An event is a topic

Choreography needs no new concept. A domain event — tenant:created, order:shipped — is a topic like any other, and a reaction to it is an ordinary message handler. The only thing that makes it an "event" rather than a "command" is intent: nobody is waiting for a result, and the emitter does not know or care who handles it.

Emitting an event

An emitter publishes fire-and-forget through the same outbound sender the two-tier calls use — a topic and a payload, with a Void response type so the caller does not wait:

(informative, .NET)

// Fire-and-forget: Void response type => no wait, no reply.
await sender.SendAsync<TenantCreated, Void>("tenant:created", new TenantCreated { TenantId = id });

Where tenant:created goes is a routing-table entry, declared once at startup — SNS for fan-out, EventBridge for bus-routed events, SQS for a point-to- point queue:

services.AddOutboundRouting(routing => routing
    .Route("tenant:created", pipeline => pipeline.UseSns(tenantEventsTopicArn))
    // or onto an EventBridge bus, where the topic becomes the event's detail-type:
    .Route("order:shipped",  pipeline => pipeline.UseEventBridge(source: "orders", eventBusName: "domain")));

Benzene tags the message so a Benzene consumer can route on it with no extra configuration:

Reacting to an event

A reacting service mounts the matching inbound transport and writes a handler for the event topic — identical in shape to a command handler, because to Benzene it is one:

app.UseAwsLambda(events => events
    .UseSns(sns => sns.UseMessageHandlers())          // topic from the "topic" attribute
    .UseEventBridge(bus => bus.UseMessageHandlers())); // topic from detail-type

[Message("tenant:created")]
public class SendWelcomeEmailOnTenantCreated : IMessageHandler<TenantCreated>
{
    public async Task<IBenzeneResult> HandleAsync(TenantCreated message) { /* react */ }
}

Adding a second reaction is adding a second handler in a second service that subscribes to the same topic. The emitter is untouched, and never learns the reaction exists — that decoupling is the whole point of choreographing.

Events from non-Benzene producers

An event from an AWS service or a non-Benzene producer will not carry Benzene's topic attribute. Two clean options:

Choreography is visible in the mesh — for free

Choreography's classic drawback is that the flow lives nowhere: no orchestrator to read, so "what reacts to what" is folklore. On Benzene it is not folklore. A consumer stamps traceparent from the inbound event's span, and the mesh derives consumer edges from that trace parentage — an event whose parent span belongs to another service makes this service a consumer of that topic, never declared. Keep trace-context propagation on (it is a Cloud Service Profile requirement) and the choreography graph draws itself in the fleet view: every emitter, every reaction, observed from real traffic. The thing that makes choreography hard to see everywhere else, Benzene gives you as a live diagram.

Reliability: at-least-once, so make reactions idempotent

SNS, SQS, and EventBridge are at-least-once — a reaction can be delivered more than once (a redelivery, a retry, a duplicate publish). Choreographed reactions must therefore be idempotent: processing the same event twice must have the same effect as once.

Benzene ships the seam for this — Benzene.Idempotency. Add UseIdempotency() to a reaction's pipeline and it derives a key (an idempotency-key header if present, else a hash of topic+body), atomically claims it in a store, and runs the handler only on the first sighting:

sns.UseIdempotency().UseMessageHandlers();

The in-memory store is single-process; for a fleet of Lambdas you supply an IIdempotencyStore backed by an atomic conditional write (DynamoDB attribute_not_exists, Redis SET NX). Idempotent reactions are what make at-least-once safe — and they are the same discipline the outbox relies on downstream, and that a saga's compensations already require.

There is no automatic cross-service rollback in choreography. If a reaction fails, it retries (and eventually dead-letters) on its own — the emitter has already moved on. Choreograph reactions that are safe to retry independently; keep anything that must be undone-as-a-unit inside an orchestrated saga.

Events are contracts

An emitted event is a published contract other services depend on — evolve it as carefully as a request type. Version an event's payload with payload-schema versioning so old consumers keep reading it, and give a breaking event a topic version rather than redefining the old topic under everyone's feet.

Checklist

Choreography is well-formed when:

See also: transactional outbox (so the events you choreograph on are never lost), and CQRS & read models (a major consumer of domain events).