orchestration saga pattern

Orchestration Saga Pattern With Spring Boot

Overview:

In this tutorial, I would like to show you a simple implementation of Orchestration Saga Pattern with Spring Boot.

Over the years, Microservices have become very popular. Microservices are distributed systems. They are smaller, modular, easy to deploy and scale etc. Developing a single Microservice application might be interesting! But handling a business transaction which spans across multiple Microservices is not fun!  In order to complete an application workflow / a task, multiple Microservices might have to work together.

Let’s see how difficult it could be in dealing with transactions / data consistency in the distributed systems in this article & how Orchestration Saga Pattern could help us.

A Simple Transaction:

Let’s assume that our business rule says, when a user places an order, order will be fulfilled if the product’s price is within the user’s credit limit/balance & the inventory is available for the product. Otherwise it will not be fulfilled. It looks really simple. This is very easy to implement in a monolith application. The entire workflow can be considered as 1 single transaction. It is easy to commit / rollback when everything is in a single DB. With distributed systems with multiple databases, It is going to be very complex! Let’s look at our architecture first to see how to implement this.

We have below Microservices with its own DB.

  • order-service
  • payment-service
  • inventory-service

When the order-service receives the request for the new order, It has to check with the payment-service & inventory-service. We deduct the payment, inventory and fulfill the order finally! What will happen if we deducted payment but if inventory is not available? How to roll back? It is difficult when multiple databases are involved.

orchestration saga pattern

Saga Pattern:

Each business transaction which spans multiple microservices are split into micro-service specific local transactions and they are executed in a sequence to complete the business workflow. It is called Saga. It can be implemented in 2 ways.

  • Choreography approach
  • Orchestration approach

In this article, we will be discussing the Orchestration based saga. For more information on Choreography based saga, check here.

Orchestration Saga Pattern:

In this pattern, we will have an orchestrator, a separate service, which will be coordinating all the transactions among all the Microservices. If things are fine, it makes the order-request as complete, otherwise marks that as cancelled.

Let’s see how we could implement this. Our sample architecture will be more or less like this.!

  • In this demo, communication between orchestrator and other services would be a simple HTTP in a non-blocking asynchronous way to make this stateless.
  • We can also use Kafka topics for this communication. For that we have to use scatter/gather pattern which is more of a stateful style.

orchestration saga pattern

Common DTOs:

  • First I create a Spring boot multi module maven project as shown below.

  • I create common DTOs/models which will be used across all the microservices. (I would suggest you to follow this approach for DTOs)

Inventory Service:

Each microservice which will be coordinated by orchestrator is expected to have at least 2 endpoints for each entity! One is deducting and other one is for resetting the transaction. For example. if we deduct inventory first and then later when we come to know that insufficient balance from payment system, we need to add the inventory back.

Note: I used a map as a DB to hold some inventory for few product IDs.

@Service
public class InventoryService {

    private Map<Integer, Integer> productInventoryMap;

    @PostConstruct
    private void init(){
        this.productInventoryMap = new HashMap<>();
        this.productInventoryMap.put(1, 5);
        this.productInventoryMap.put(2, 5);
        this.productInventoryMap.put(3, 5);
    }

    public InventoryResponseDTO deductInventory(final InventoryRequestDTO requestDTO){
        int quantity = this.productInventoryMap.getOrDefault(requestDTO.getProductId(), 0);
        InventoryResponseDTO responseDTO = new InventoryResponseDTO();
        responseDTO.setOrderId(requestDTO.getOrderId());
        responseDTO.setUserId(requestDTO.getUserId());
        responseDTO.setProductId(requestDTO.getProductId());
        responseDTO.setStatus(InventoryStatus.UNAVAILABLE);
        if(quantity > 0){
            responseDTO.setStatus(InventoryStatus.AVAILABLE);
            this.productInventoryMap.put(requestDTO.getProductId(), quantity - 1);
        }
        return responseDTO;
    }

    public void addInventory(final InventoryRequestDTO requestDTO){
        this.productInventoryMap
                .computeIfPresent(requestDTO.getProductId(), (k, v) -> v + 1);
    }

}
  • controller
@RestController
@RequestMapping("inventory")
public class InventoryController {

    @Autowired
    private InventoryService service;

    @PostMapping("/deduct")
    public InventoryResponseDTO deduct(@RequestBody final InventoryRequestDTO requestDTO){
        return this.service.deductInventory(requestDTO);
    }

    @PostMapping("/add")
    public void add(@RequestBody final InventoryRequestDTO requestDTO){
        this.service.addInventory(requestDTO);
    }

}

Payment Service:

It also exposes 2 endpoints like inventory-service. I am showing only the important classes. For more details please check the github link at the end of this article for the complete project source code.

@Service
public class PaymentService {

    private Map<Integer, Double> userBalanceMap;

    @PostConstruct
    private void init(){
        this.userBalanceMap = new HashMap<>();
        this.userBalanceMap.put(1, 1000d);
        this.userBalanceMap.put(2, 1000d);
        this.userBalanceMap.put(3, 1000d);
    }

    public PaymentResponseDTO debit(final PaymentRequestDTO requestDTO){
        double balance = this.userBalanceMap.getOrDefault(requestDTO.getUserId(), 0d);
        PaymentResponseDTO responseDTO = new PaymentResponseDTO();
        responseDTO.setAmount(requestDTO.getAmount());
        responseDTO.setUserId(requestDTO.getUserId());
        responseDTO.setOrderId(requestDTO.getOrderId());
        responseDTO.setStatus(PaymentStatus.PAYMENT_REJECTED);
        if(balance >= requestDTO.getAmount()){
            responseDTO.setStatus(PaymentStatus.PAYMENT_APPROVED);
            this.userBalanceMap.put(requestDTO.getUserId(), balance - requestDTO.getAmount());
        }
        return responseDTO;
    }

    public void credit(final PaymentRequestDTO requestDTO){
        this.userBalanceMap.computeIfPresent(requestDTO.getUserId(), (k, v) -> v + requestDTO.getAmount());
    }

}
  • controller
@RestController
@RequestMapping("payment")
public class PaymentController {

    @Autowired
    private PaymentService service;

    @PostMapping("/debit")
    public PaymentResponseDTO debit(@RequestBody PaymentRequestDTO requestDTO){
        return this.service.debit(requestDTO);
    }

    @PostMapping("/credit")
    public void credit(@RequestBody PaymentRequestDTO requestDTO){
        this.service.credit(requestDTO);
    }

}

Order Service:

Our order service receives the create order command and raises an order-created event using spring boot kafka binder. It also listens to order-updated channel/kafka topic and updates order status.

  • controller
@RestController
@RequestMapping("order")
public class OrderController {

    @Autowired
    private OrderService service;

    @PostMapping("/create")
    public PurchaseOrder createOrder(@RequestBody OrderRequestDTO requestDTO){
        requestDTO.setOrderId(UUID.randomUUID());
        return this.service.createOrder(requestDTO);
    }

    @GetMapping("/all")
    public List<OrderResponseDTO> getOrders(){
        return this.service.getAll();
    }

}
  • service
@Service
public class OrderService {

    // product price map
    private static final Map<Integer, Double> PRODUCT_PRICE =  Map.of(
            1, 100d,
            2, 200d,
            3, 300d
    );

    @Autowired
    private PurchaseOrderRepository purchaseOrderRepository;

    @Autowired
    private FluxSink<OrchestratorRequestDTO> sink;

    public PurchaseOrder createOrder(OrderRequestDTO orderRequestDTO){
        PurchaseOrder purchaseOrder = this.purchaseOrderRepository.save(this.dtoToEntity(orderRequestDTO));
        this.sink.next(this.getOrchestratorRequestDTO(orderRequestDTO));
        return purchaseOrder;
    }

    public List<OrderResponseDTO> getAll() {
        return this.purchaseOrderRepository.findAll()
                .stream()
                .map(this::entityToDto)
                .collect(Collectors.toList());
    }

    private PurchaseOrder dtoToEntity(final OrderRequestDTO dto){
        PurchaseOrder purchaseOrder = new PurchaseOrder();
        purchaseOrder.setId(dto.getOrderId());
        purchaseOrder.setProductId(dto.getProductId());
        purchaseOrder.setUserId(dto.getUserId());
        purchaseOrder.setStatus(OrderStatus.ORDER_CREATED);
        purchaseOrder.setPrice(PRODUCT_PRICE.get(purchaseOrder.getProductId()));
        return purchaseOrder;
    }

    private OrderResponseDTO entityToDto(final PurchaseOrder purchaseOrder){
        OrderResponseDTO dto = new OrderResponseDTO();
        dto.setOrderId(purchaseOrder.getId());
        dto.setProductId(purchaseOrder.getProductId());
        dto.setUserId(purchaseOrder.getUserId());
        dto.setStatus(purchaseOrder.getStatus());
        dto.setAmount(purchaseOrder.getPrice());
        return dto;
    }

    public OrchestratorRequestDTO getOrchestratorRequestDTO(OrderRequestDTO orderRequestDTO){
        OrchestratorRequestDTO requestDTO = new OrchestratorRequestDTO();
        requestDTO.setUserId(orderRequestDTO.getUserId());
        requestDTO.setAmount(PRODUCT_PRICE.get(orderRequestDTO.getProductId()));
        requestDTO.setOrderId(orderRequestDTO.getOrderId());
        requestDTO.setProductId(orderRequestDTO.getProductId());
        return requestDTO;
    }

}

Order Orchestrator:

This is a microservice which is responsible for coordinating all the transactions. It listens to order-created topic. As and when a new order is created, It immediately builds separate request to each service like payment-service/inventory-service etc and validates the responses. If they are OK, fulfills the order. If one of them is not, cancels the oder. It also tries to reset any of local transactions which happened in any of the microservices.

We consider all the local transactions as 1 single workflow. A workflow will contain multiple workflow steps.

  • Workflow step
public interface WorkflowStep {

    WorkflowStepStatus getStatus();
    Mono<Boolean> process();
    Mono<Boolean> revert();

}
  • Workflow
public interface Workflow {

    List<WorkflowStep> getSteps();

}
  • In our case, for the Order workflow, we have 2 steps. Each implementation should know how to do local transaction and how to reset.
  • Inventory step
public class InventoryStep implements WorkflowStep {

    private final WebClient webClient;
    private final InventoryRequestDTO requestDTO;
    private WorkflowStepStatus stepStatus = WorkflowStepStatus.PENDING;

    public InventoryStep(WebClient webClient, InventoryRequestDTO requestDTO) {
        this.webClient = webClient;
        this.requestDTO = requestDTO;
    }

    @Override
    public WorkflowStepStatus getStatus() {
        return this.stepStatus;
    }

    @Override
    public Mono<Boolean> process() {
        return this.webClient
                .post()
                .uri("/inventory/deduct")
                .body(BodyInserters.fromValue(this.requestDTO))
                .retrieve()
                .bodyToMono(InventoryResponseDTO.class)
                .map(r -> r.getStatus().equals(InventoryStatus.AVAILABLE))
                .doOnNext(b -> this.stepStatus = b ? WorkflowStepStatus.COMPLETE : WorkflowStepStatus.FAILED);
    }

    @Override
    public Mono<Boolean> revert() {
        return this.webClient
                    .post()
                    .uri("/inventory/add")
                    .body(BodyInserters.fromValue(this.requestDTO))
                    .retrieve()
                    .bodyToMono(Void.class)
                    .map(r ->true)
                    .onErrorReturn(false);
    }
}
  • Payment step
public class PaymentStep implements WorkflowStep {

    private final WebClient webClient;
    private final PaymentRequestDTO requestDTO;
    private WorkflowStepStatus stepStatus = WorkflowStepStatus.PENDING;

    public PaymentStep(WebClient webClient, PaymentRequestDTO requestDTO) {
        this.webClient = webClient;
        this.requestDTO = requestDTO;
    }

    @Override
    public WorkflowStepStatus getStatus() {
        return this.stepStatus;
    }

    @Override
    public Mono<Boolean> process() {
        return this.webClient
                    .post()
                    .uri("/payment/debit")
                    .body(BodyInserters.fromValue(this.requestDTO))
                    .retrieve()
                    .bodyToMono(PaymentResponseDTO.class)
                    .map(r -> r.getStatus().equals(PaymentStatus.PAYMENT_APPROVED))
                    .doOnNext(b -> this.stepStatus = b ? WorkflowStepStatus.COMPLETE : WorkflowStepStatus.FAILED);
    }

    @Override
    public Mono<Boolean> revert() {
        return this.webClient
                .post()
                .uri("/payment/credit")
                .body(BodyInserters.fromValue(this.requestDTO))
                .retrieve()
                .bodyToMono(Void.class)
                .map(r -> true)
                .onErrorReturn(false);
    }

}
  • service / coordinator
@Service
public class OrchestratorService {

    @Autowired
    @Qualifier("payment")
    private WebClient paymentClient;

    @Autowired
    @Qualifier("inventory")
    private WebClient inventoryClient;

    public Mono<OrchestratorResponseDTO> orderProduct(final OrchestratorRequestDTO requestDTO){
        Workflow orderWorkflow = this.getOrderWorkflow(requestDTO);
        return Flux.fromStream(() -> orderWorkflow.getSteps().stream())
                .flatMap(WorkflowStep::process)
                .handle(((aBoolean, synchronousSink) -> {
                    if(aBoolean)
                        synchronousSink.next(true);
                    else
                        synchronousSink.error(new WorkflowException("create order failed!"));
                }))
                .then(Mono.fromCallable(() -> getResponseDTO(requestDTO, OrderStatus.ORDER_COMPLETED)))
                .onErrorResume(ex -> this.revertOrder(orderWorkflow, requestDTO));

    }

    private Mono<OrchestratorResponseDTO> revertOrder(final Workflow workflow, final OrchestratorRequestDTO requestDTO){
        return Flux.fromStream(() -> workflow.getSteps().stream())
                .filter(wf -> wf.getStatus().equals(WorkflowStepStatus.COMPLETE))
                .flatMap(WorkflowStep::revert)
                .retry(3)
                .then(Mono.just(this.getResponseDTO(requestDTO, OrderStatus.ORDER_CANCELLED)));
    }

    private Workflow getOrderWorkflow(OrchestratorRequestDTO requestDTO){
        WorkflowStep paymentStep = new PaymentStep(this.paymentClient, this.getPaymentRequestDTO(requestDTO));
        WorkflowStep inventoryStep = new InventoryStep(this.inventoryClient, this.getInventoryRequestDTO(requestDTO));
        return new OrderWorkflow(List.of(paymentStep, inventoryStep));
    }

    private OrchestratorResponseDTO getResponseDTO(OrchestratorRequestDTO requestDTO, OrderStatus status){
        OrchestratorResponseDTO responseDTO = new OrchestratorResponseDTO();
        responseDTO.setOrderId(requestDTO.getOrderId());
        responseDTO.setAmount(requestDTO.getAmount());
        responseDTO.setProductId(requestDTO.getProductId());
        responseDTO.setUserId(requestDTO.getUserId());
        responseDTO.setStatus(status);
        return responseDTO;
    }

    private PaymentRequestDTO getPaymentRequestDTO(OrchestratorRequestDTO requestDTO){
        PaymentRequestDTO paymentRequestDTO = new PaymentRequestDTO();
        paymentRequestDTO.setUserId(requestDTO.getUserId());
        paymentRequestDTO.setAmount(requestDTO.getAmount());
        paymentRequestDTO.setOrderId(requestDTO.getOrderId());
        return paymentRequestDTO;
    }

    private InventoryRequestDTO getInventoryRequestDTO(OrchestratorRequestDTO requestDTO){
        InventoryRequestDTO inventoryRequestDTO = new InventoryRequestDTO();
        inventoryRequestDTO.setUserId(requestDTO.getUserId());
        inventoryRequestDTO.setProductId(requestDTO.getProductId());
        inventoryRequestDTO.setOrderId(requestDTO.getOrderId());
        return inventoryRequestDTO;
    }

}

I have provided only high level details here. For the complete source, check here.

Orchestration Saga Pattern – Demo:

  • Once all the services are up and running, I send a POST request to create order. I get the order created status.
    • Do note that user 1 tries to order product id 3 which costs $300
    • The user’s credit limit is $1000

  • I sent 4 requests. So 3 requests were fulfilled. Not the 4th one as the user would have only $100 left and we can not fulfill the 4th order. So the payment service would have declined.

  • The user 1 with this available balance $100, he can buy product id 1 as it costs only $100.

Summary:

We were able to successfully demonstrate the Orchestration Saga Pattern with Spring Boot. Handling transactions and maintaining data consistency among all the microservices are difficult in general. When multiple services are involved like payment, inventory, fraud check, shipping check…..etc it would be very difficult to manage such a complex workflow with multiple steps without a coordinator. By introducing a separate service for orchestration, order-service is freed up from these responsibilities.

Check the project source code here.

Learn more about Microservices Patterns.

Happy coding 🙂

 

Share This:

27 thoughts on “Orchestration Saga Pattern With Spring Boot

  1. Wonderful sir. I am your youtube and udemy subscriber. Please keep on teaching students like me. I am working as test automation engineer at Tcs Nagpur. I love your way of teaching, and especially bringing new topics out of the traditional scenarios that we face. #BigFan

  2. Hello dear,

    I have a couple of questions about your excellent post:

    I have my client API, which I have created 3 spring boot projects (independent): customer (GET /customers/{id-customer}, POST, PUT, etc), addresses (GET /{id-customer}/addresses, POST, PUT , etc) and contacts (GET /{id-customer}/contacts, POST, PUT, etc). Could you use Stream, Flux and kafka so that when I request GET /customers/{id} it returns the complete ‘customers’ information, including the ‘address’ and ‘contacts’ service information in parallel? It means that the request must wait for the data of each service to be obtained and return all the information of a customer. Do you have any suggestions or examples?
    About the display of orders, payments and products, is there a way that the response from the execution of the POST /order/create returns the ORDER_CANCELLED status? I mean does the POST request “wait” for the process to finish completely and the updated status to be sent when an error occurs? Currently the POST /order/create returns only ORDER_CREATED, and the status changes it in the background.

    1. Hi,
      You have multiple questions here.

      1. Most of the applications are more read heavy than write. So when you need customer info, you could avoid calling multiple services internally to join all the info. Because when you need to query 3 services, when one service is down, the entire application becomes kind of down! not accessible. There is a separate pattern for this. Event carried state transfer – Check this one.

      2. If you check the screenshots, order-service always keeps the order in pending status first as soon as it gets a request. The orchestraor service is the one which coordinates all the activities and decides it can be fulfilled or cancelled. So, order-service does not wait. It is asynchronous.
        Thanks.

  3. Hi Vins,

    I followed your blog for really long time, and I saw that you had several topic related to Kafka. I am just curious about how do you perform the testing for streaming application using Kafka. May you create a post about each testing level for Kafka?

    Many thanks in advanced.

  4. after reading a dozen stuff over the internet and tried and got tired. This is one of the most satisfying among all other. and thank you very much for all the explanation. Loved it. Looking forward to read some more interesting stuff.

  5. Awesome tutorial. I struggled a lot to find distributed transaction examples and tutorials. This article is my destination!

  6. I wonder, what will happend when during the reverting process – let’s say after revert payment but before revert inventory (or vice versa) – the coordinator failed and go down. With this implementation we will leave our system in inconsistent state.

    1. Yes, It could happen. For that we could we use Kafka topics for the communication between the orchestrator and other services.
      Or this orchestrator will be maintaining its own DB for the state. That is – It has to get the confirmation for the reverting process. Otherwise it will retry.

  7. Is this orchestration approach recommended if we have one service calling 7-8 micro services?

    Also, we could call the inventory and payment service using Test template. Is a message broker or Kafka mandatory?

    Could you please let me know? Thanks

    1. Each approach has its own advantage!
      RestTemplate is simple and easy. Using message brokers helps with async processing and calling service does not have to worry about service unavailability. I would not say it is recommended. It just a demo wherever this pattern will be useful.

  8. Hello vIns
    Nice article.
    I am just wondering where in the code you invoke the orchestration service. If you could direct me to that code snippet?
    Thanks

    Raj

  9. Hi vIns
    excellent article.
    I have a question.
    I Can implement the project dont using Kafka? only calling the order-orchestration service with a HTTP request?

  10. Hi,
    I see your current code don’t return correct “status”: “ORDER_CANCELLED” on 4th attempts. Could you please double check?

    1. Thanks a lot. Looks like it broke with recent version of Spring. I have updated the code as few things got deprecated as well. It seems to work now.

      1. Hey, I do see code still not working with the latest code, I double check many times. Something weird happening. Would you want to double check again ?

        1. I think there is confusion with the Status – it always return ORDER_CREATED irrespective of success or failure. Its happening because you’ve hardcoded in OrderService.java dtoToEntity method need some correction. Please do the needful and inform back.

          1. Yes, it is ORDER CREATED. Not Completed or Failed. The status will get updated eventually!!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.