For technical leaders exploring event-driven systems versus workflow engines, this post builds on our earlier discussion of bespoke process databases and highlights why choreography struggles in high-volume, cross-domain environments.
This post is part-2 of my initial post Why Event-Driven Architecture Creates Bespoke Process Databases – and How Workflow Engines like Temporal Solve It
Introduction
Event-driven systems have become the backbone of modern architectures. They enable services to react independently to events, scale autonomously, and remain loosely coupled. This approach, known as choreography, allows business processes to emerge from the collective reaction of services without a central controller.
But as event-driven architectures grow in scale and complexity — for example, 10 million daily events, where each event may need to invoke 7 to 12 dependent services across domains such as orders, payments, subscriptions, customer onboarding, or customer profiles based on subscribed plans — the choreography model starts to break down. Business logic turns into “event spaghetti,” debugging becomes a nightmare, and resiliency depends on fragmented, custom implementations.
An alternative is workflow orchestration: using a central workflow engine to direct the sequence of steps, manage state, and ensure reliability across domains. Instead of leaving coordination to scattered services, orchestration provides a single source of truth for process execution, bringing order where choreography collapses.
The Choreography Problem at Scale
1. Event Spaghetti
When millions of daily events ripple through 7 to 12 interdependent services across domains, the process turns into what the industry often calls “event spaghetti.”
The metaphor comes from “spaghetti code” — code that is tangled and hard to follow, like a plate of noodles. In the same way, event spaghetti means:
-
Business logic spread across orders, payments, subscriptions, onboarding, and customer profiles
-
Events bouncing around unpredictably between services, sometimes looping or arriving out of order – it means that at scale, across multiple services, partitions, and domains, events that belong to the same business process may not line up neatly in sequence unless you carefully enforce partitioning and correlation.
-
If your design ensures all events of a single transaction (say an order lifecycle) use the same partition key, then you’ll get strict ordering for that transaction.
-
-
Dependencies overlapping without clear ownership – means many services rely on the same event, but nobody owns the full business process those dependencies collectively support.
In simpler English, you can think of this as “a tangled mess of events.”
A single failed transaction requires piecing together logs and messages across multiple domains, APIs, and data stores — and depending on the payload design, there may or may not be a common business transaction ID that ties all related objects together. Without a consistent ID strategy, tracing the full flow becomes guesswork.
The Correlation ID Problem
In theory, a business transaction ID should travel with every event, linking all related objects (order, payment, subscription, profile update) to a single business process. In practice:
-
Some teams include IDs, others don’t
-
IDs may change form (orderId, paymentId, subscriptionId) without a universal mapping
-
Events from external systems often arrive with missing or inconsistent identifiers
The result: even if all the right events exist, there’s no reliable way to stitch them together into a coherent story.
Workflow engines solve this by design. Each workflow instance has a unique, durable ID that acts as the correlation anchor for the entire business process. Regardless of how many steps, retries, or domains are involved, the workflow ID provides a consistent reference point — eliminating the guesswork of chasing correlation IDs across scattered payloads.
2. Complex State Management
Choreography forces each domain team to build its own persistence and retry logic. At this scale:
-
Payments build retry queues for failed authorizations
-
Subscriptions track pending plan activations
-
Onboarding manages incomplete KYC or document submissions
-
Customer profiles hold partial state while waiting for upstream signals
The result: 7 to 12 mini workflow engines scattered across domains, none of which provide a reliable end-to-end view.
3. Debugging Nightmares
When something goes wrong:
-
There is no single traceable thread from order creation to final fulfillment
-
Each IT director owns a slice, but no one owns the full picture
-
Debugging requires chasing events across asynchronous systems, inconsistent APIs, and siloed logs
4. Weak Resiliency
Without orchestration, resiliency depends on each team’s custom implementation.
-
Payments retry logic might differ from subscriptions
-
Onboarding APIs may respond in different formats than profiles
-
A lost or delayed event in one domain can stall the entire business process indefinitely
How Workflow Engines Change the Equation
Workflow engines such as Temporal, Cadence, or Camunda provide orchestration. Instead of scattering process logic across 7 to 12 domain teams, the workflow engine becomes the durable, authoritative source of truth for cross-domain processes.
They address the challenges by:
-
Visibility: A single, queryable state for each in-flight process, regardless of how many domains are involved
-
State management: Durable persistence, retries, and timeouts without custom logic per team
-
Consistency: Standardized execution paths even when underlying APIs are inconsistent
-
Resiliency: Built-in compensations and recovery ensure processes complete reliably, even if one domain lags or fails
Choreography vs. Orchestration: When to Use Each
Choreography isn’t inherently bad. It works well for:
-
Lightweight, stateless reactions (logging, analytics, notifications)
-
Independent updates within a single domain
But orchestration shines when processes:
-
Span multiple business domains (orders, payments, subscriptions, onboarding, profiles)
-
Involve 7 to 12 interdependent steps per event
-
Are governed by different IT directors with inconsistent APIs
-
Require end-to-end reliability and visibility
The Takeaway
Choreography may work in small or single-domain systems, but at scale — with 10 million daily events spanning 7 to 12 dependencies across multiple business domains managed by different IT directors — it collapses into event spaghetti (a tangled mess of events), fragmented state management, debugging nightmares, and weak resiliency.
Workflow engines bring order to this chaos by centralizing process control, standardizing state management, and ensuring durability by design.
For technical leaders, the lesson is clear:
-
Use choreography for simple, domain-contained event reactions
-
Use orchestration (via workflow engines) for cross-domain, business-critical processes where reliability and visibility matter most
Connecting the Series
This post is the second in a series on the limitations of event-driven architectures.
-
In Part 1, we explored why event-driven systems often lead to bespoke process databases as teams try to handle state, retries, and compensations on their own.
-
In this Part 2, we examined why event-driven choreography itself breaks down at scale, and how workflow orchestration provides a cleaner alternative.
Together, these posts highlight a central theme: event-driven systems are powerful, but without orchestration, complexity eventually leaks into databases, APIs, and operations. Workflow engines solve that by making process control explicit, durable, and reliable.