Benzene

Event-Driven Choreography

Status: DRAFT v0.1 — part of the Patterns "composing a system" set.

The two-tier architecture drives processes with orchestrators: a central service issues commands to core services and runs a saga to keep the multi-service write atomic. Choreography is the other way to move a process across a fleet: no central conductor — each service reacts to events and emits its own, and the process emerges from the chain of reactions.

Both are first-class on Benzene, and a real system uses both. This document is about when to choreograph, and exactly how you build it with Benzene's events, transports, and mesh.

Orchestration vs. choreography

Orchestration (orchestrators.md) Choreography (this doc)
Control Central — one service directs the process Distributed — each service decides its own reaction
Coupling Orchestrator knows every step Emitter knows nothing about who reacts
Atomicity A saga: all-or-nothing across services None built-in — each reaction succeeds/retries on its own
Adding a step Edit the orchestrator Add a new consumer; no existing service changes
Best for An invariant that must hold across services (a signup that must not half-apply) Fan-out reactions that are independently retryable (on signup: send a welcome email, warm a cache, start a trial clock)

The rule of thumb: orchestrate what must be atomic; choreograph what must merely happen. A process usually has both — a small atomic core (orchestrated as a saga) that emits an event, and a spray of independent reactions (choreographed). Do not use choreography to hold a cross-service invariant; there is no central rollback, and reconstructing one out of scattered compensations is exactly the complexity the saga exists to remove.

How you build it with Benzene

An event is a topic

Choreography needs no new concept. A domain event — tenant:created, order:shipped — is a topic like any other, and a reaction to it is an ordinary message handler. The only thing that makes it an "event" rather than a "command" is intent: nobody is waiting for a result, and the emitter does not know or care who handles it.

Emitting an event

An emitter publishes fire-and-forget through the same outbound sender the two-tier calls use — a topic and a payload, with a Void response type so the caller does not wait:

(informative, .NET)

// Fire-and-forget: Void response type => no wait, no reply.
await sender.SendAsync<TenantCreated, Void>("tenant:created", new TenantCreated { TenantId = id });

Where tenant:created goes is a routing-table entry, declared once at startup — SNS for fan-out, EventBridge for bus-routed events, SQS for a point-to- point queue:

services.AddOutboundRouting(routing => routing
    .Route("tenant:created", pipeline => pipeline.UseSns(tenantEventsTopicArn))
    // or onto an EventBridge bus, where the topic becomes the event's detail-type:
    .Route("order:shipped",  pipeline => pipeline.UseEventBridge(source: "orders", eventBusName: "domain")));

Benzene tags the message so a Benzene consumer can route on it with no extra configuration:

Reacting to an event

A reacting service mounts the matching inbound transport and writes a handler for the event topic — identical in shape to a command handler, because to Benzene it is one:

app.UseAwsLambda(events => events
    .UseSns(sns => sns.UseMessageHandlers())          // topic from the "topic" attribute
    .UseEventBridge(bus => bus.UseMessageHandlers())); // topic from detail-type

[Message("tenant:created")]
public class SendWelcomeEmailOnTenantCreated : IMessageHandler<TenantCreated>
{
    public async Task<IBenzeneResult> HandleAsync(TenantCreated message) { /* react */ }
}

Adding a second reaction is adding a second handler in a second service that subscribes to the same topic. The emitter is untouched, and never learns the reaction exists — that decoupling is the whole point of choreographing.

Events from non-Benzene producers

An event from an AWS service or a non-Benzene producer will not carry Benzene's topic attribute. Two clean options:

Choreography is visible in the mesh — for free

Choreography's classic drawback is that the flow lives nowhere: no orchestrator to read, so "what reacts to what" is folklore. On Benzene it is not folklore. A consumer stamps traceparent from the inbound event's span, and the mesh derives consumer edges from that trace parentage — an event whose parent span belongs to another service makes this service a consumer of that topic, never declared. Keep trace-context propagation on (it is a Cloud Service Profile requirement) and the choreography graph draws itself in the fleet view: every emitter, every reaction, observed from real traffic. The thing that makes choreography hard to see everywhere else, Benzene gives you as a live diagram.

Reliability: at-least-once, so make reactions idempotent

SNS, SQS, and EventBridge are at-least-once — a reaction can be delivered more than once (a redelivery, a retry, a duplicate publish). Choreographed reactions must therefore be idempotent: processing the same event twice must have the same effect as once.

Benzene ships the seam for this — Benzene.Idempotency. Add UseIdempotency() to a reaction's pipeline and it derives a key (an idempotency-key header if present, else a hash of topic+body), atomically claims it in a store, and runs the handler only on the first sighting:

sns.UseIdempotency().UseMessageHandlers();

The in-memory store is single-process; for a fleet of Lambdas you supply an IIdempotencyStore backed by an atomic conditional write (DynamoDB attribute_not_exists, Redis SET NX). Idempotent reactions are what make at-least-once safe — and they are the same discipline the outbox relies on downstream, and that a saga's compensations already require.

There is no automatic cross-service rollback in choreography. If a reaction fails, it retries (and eventually dead-letters) on its own — the emitter has already moved on. Choreograph reactions that are safe to retry independently; keep anything that must be undone-as-a-unit inside an orchestrated saga.

Events are contracts

An emitted event is a published contract other services depend on — evolve it as carefully as a request type. Version an event's payload with payload-schema versioning so old consumers keep reading it, and give a breaking event a topic version rather than redefining the old topic under everyone's feet.

Checklist

Choreography is well-formed when:

See also: transactional outbox (so the events you choreograph on are never lost), and CQRS & read models (a major consumer of domain events).