Implementing the Saga Pattern for Distributed Transactions in Microservices

The Problem: Distributed Checkout Gone Wrong

When I joined Full Truck Alliance, the logistics trading platform was handling 500K+ daily transactions -- but the checkout process was a mess. Five different microservices (cargo, orders, contracts, waybill, payment) each managed their own data, and when one failed mid-transaction, the others had no idea.

The result? Inconsistent order states, stuck payments, and a support team drowning in tickets.

Why Not Two-Phase Commit?

The first instinct for distributed transactions is often the two-phase commit (2PC) protocol. But in a microservices architecture with high throughput, 2PC has fatal flaws:

• Blocking: All participants lock resources until the coordinator decides
• Single point of failure: If the coordinator crashes, everything stalls
• Poor scalability: Doesn't work across different database technologies

For a system processing half a million transactions daily, we needed something better.

Enter the Saga Pattern

The Saga pattern breaks a distributed transaction into a sequence of local transactions. Each service completes its step and publishes an event. If any step fails, compensating transactions undo the previous steps.

We chose the orchestration approach over choreography for several reasons:

1. Centralized control: A single orchestrator manages the workflow logic
2. Easier debugging: One place to track the entire transaction state
3. Simpler error handling: Compensating actions are defined alongside forward actions

Implementation Architecture

The Saga Orchestrator

We built a lightweight orchestrator service that defined each checkout saga as a state machine:

java

public class CheckoutSaga {
    private final SagaDefinition<CheckoutContext> sagaDefinition;

    public CheckoutSaga() {
        this.sagaDefinition = SagaDefinition.<CheckoutContext>builder()
            .step("reserveCargo")
                .invoke(this::reserveCargo)
                .withCompensation(this::releaseCargo)
            .step("createOrder")
                .invoke(this::createOrder)
                .withCompensation(this::cancelOrder)
            .step("generateContract")
                .invoke(this::generateContract)
                .withCompensation(this::voidContract)
            .step("createWaybill")
                .invoke(this::createWaybill)
                .withCompensation(this::cancelWaybill)
            .step("processPayment")
                .invoke(this::processPayment)
                .withCompensation(this::refundPayment)
            .build();
    }
}

Message Queue Integration

Each step communicates via RabbitMQ, ensuring reliable delivery even if a service is temporarily down. The orchestrator publishes commands, and services respond with results or failure events.

Idempotency Is Non-Negotiable

Every service handler is idempotent. If a message is delivered twice (which WILL happen in distributed systems), the result is the same. We used Redis-based deduplication keys with a 24-hour TTL.

Results

After rolling out the Saga-based checkout:

• 40% reduction in abnormal order status incidents
• Zero data inconsistency issues in payment settlements
• Faster recovery: When failures occur, compensating transactions execute in under 2 seconds
• Better observability: Complete transaction traces in our monitoring dashboard

Key Takeaways for Your Projects

1. Start with the failure scenarios: Design compensating transactions before the happy path
2. Make everything idempotent: Assume every message will be delivered at least twice
3. Use correlation IDs: Track the entire saga with a single ID across all services
4. Test failures aggressively: Use chaos engineering to verify compensating transactions work
5. Monitor saga state: Build dashboards showing in-progress, completed, and compensating sagas

The Saga pattern isn't the answer to every distributed transaction problem, but for high-throughput systems where availability matters more than strict consistency, it's a battle-tested approach.