Examples: Asynchronous communication between microservices with Kafka

Mon, 01 Jan 0001 00:00:00 +0000

DevPrime-based applications natively incorporate an event-driven architecture approach. They receive external events, process business rules, emit internal events, and, when necessary, propagate the events externally using the Stream adapter, which connects natively to Kafka, Kafka, and other platforms.

Now, we’ll put together a demo involving two microservices. The first, “Order”, will receive orders, while the second, “Payment”, will process payments. In the example, we’ll use two containers built on docker. It is essential to upload MongoDB and Kafka containers.

If you already have the MongoDB and Kafka services locally, change the settings in the /src/App/app/appsettings.json file in the State and Stream sections with your credentials.

Checklist and preperation of the initial environment:

Open an account on the Devprime platform and purchase a [(Developer/Enterprise)] license(https://devprime.io/pricing).
Install an updated version of .NET (Linux, macOS, and Windows)
Install and/or update Visual Studio Code and/or Visual Studio 2023 Community / Professional / Enterprise.
Install and/or update docker (For Windows, use WSL2).
Initialize the MongoDB and Kafka containers on docker and add the ‘orderevents’ and ‘paymentevents’ queue in Kafka.
Install and activate the latest version of the Devprime CLI.
Create a folder for your projects and set read and write permissions.
See the article “Creating the First Microservice” to explore getting started with Devprime.

Sample source code with the Order and Payment microservices

This article will implement the sample code below for all Devprime platform subscribers. You don’t need to download it and you can follow the implementation in the next topics.

`1`	`git clone https://github.com/devprime/devprime-microservices-order-payment`

Important: If you have downloaded the project, you need to run the command dp stack in the cloned folder to update your Devprime license in the projects.

`1`	`dp stack`

Creating the first “Order” microservice

We will use the DevPrime CLI for creating the microservices. In this example, we’ll enter the name of the new application, the Stream service type as “Kafka” for asynchronous event emitting, and the State as “MongoDB” for data persistence. To start a new microservice, use the sample command that appears. A new application will be created in seconds and will be ready for production.
dp new Order --stream kafka --state mongodb

Adding an Order Business Rule via Devprime Marketplace
dp marketplace order

Running accelerator for implementing code based on the Domain. When you run the command for the first time in this demonstration, type “A” so that you can move forward and make the changes.
dp init

The accelerator automatically implements the classes of “Domain Events” in Domain, “Event Handlers” in Application, “Domain Services” in Application, “Application Services” in Application, persistence in State, APIs in Web and some unit tests in Tests as examples.

When you run the application again using the “.\run.ps1” (Windows) or “.\run.sh” (Linux/macOS) script we will already have a new view of your API. Open the web browser at the url http://localhost:5000 and locate the post method in Swagger and click “Try it out”.

Fill in the JSON data and then click “execute”. Return to the command prompt and track the result in the microservice log automatically generated by the Devprime Observability adapter.

To check out a code example implemented by the accelerator, follow the code in the src/Core/Application/EventHandlers/Order/OrderCreatedEventHandler.cs folder that notifies via the Stream adapter.

code src/Core/Application/EventHandlers/Order/CreateOrderEventHandler.cs

public override dynamic Handle(OrderCreated orderCreated)
    {
        var success = false;
        var order = orderCreated.Get<Domain.Aggregates.Order.Order>();
        var destination = Dp.Settings.Default("stream.orderevents");
        var eventName = "OrderCreated";
        var eventData = new OrderCreatedEventDTO()
        {
            ID = order.ID,
            CustomerName = order.CustomerName,
            CustomerTaxID = order.CustomerTaxID,
            Total = order.Total
        };
        Dp.Stream.Send(destination, eventName, eventData);
        success = true;
        return success;
    }

At this point, the log demonstrates the receipt of an HTTP request in the “Order” API, which passes it on to the application service. Then, the business rule in the domain is processed, emitting the “OrderCreated” event. This event has its state persisted in MongoDB and results in a notification in the external world by the Stream.

When accessing Kafka through the portal, you can observe this event in the “OrderEvents” queue. This means that the “Order” microservice has already fulfilled its role and notified this business fact so that it can be noticed by another service.

Since the communication in this example is fully asynchronous, the events in the Kafka queue will be read by the “Payment” microservice as configured. You can stop the application by pressing “Control” + “C” at any time and load it again when you want.

Congratulations !! You already have the “Order” microservice

Creating the second microservice “Payment”

The second microservice will be called “Payment”, and we’ll start building it at this point. During implementation, an additional item that we will configure will be the activation of “subscribe” on the Stream adapter, allowing the “Payment” microservice to receive the events emitted by the “Order” microservice.

Running dp new and building microservices
dp new ms-payment --state mongodb --stream kafka

Adding a Payment Business Rule via Devprime Marketplace
dp marketplace payment

Running Devprime’s “dp init” command for implementing Domain-Driven Design (DDD)-based code.
dp init

To run both microservices in the same on-premises environment, you need to change the HTTP port of the “Payment” microservice. In the “ms-payment” project folder, edit the file src/App/appsettings.json, in the “web” key, changing it to 5002 and 5003, as shown in the example below:

  "Devprime_Web": {
    "url": "https://localhost:5003;http://localhost:5002",
    "enable": "true",
    "enableswagger": "true",
    "PostSuccess": "201",
    "PostFailure": "500",
    "GetSuccess": "200",
    "GetFailure": "500",
    "PatchSuccess": "200",
    "PatchFailure": "500",
    "PutSuccess": "200",
    "PutFailure": "500",
    "DeleteSuccess": "200",
    "DeleteFailure": "500"
  },

Run the application using the command below
.\run.ps1 ou ./run.sh (Linux, MacOS)

If the application has run, everything is fine. Close it with “Control+C” and we will follow up with the implications to enable the reception of events emitted by the “Order” microservices.

The next step is to add the configuration in the Stream service and create a DTO to receive the event data. You can do the whole process manually or by using the accelerator below.
dp add subscribe OrderCreated -as PaymentService

When you confirm the change, you will have three modified files so that we can move forward in the settings.

1
2
3

src/Core/Application/Services/Payment/Model/OrderCreatedEventDTO.cs
src/Adapters/Stream/EventStream.cs
src/Adapters/Stream/GlobalUsings.cs

Open the OrderCreatedEventDTO.cs file and add the following properties:

Visual Studio Code
code src/Core/Application/Services/Payment/Model/OrderCreatedEventDTO.cs

public class OrderCreatedEventDTO                     
  {                                                     
    public Guid OrderID { get; set; }
    public string CustomerName { get; set; }
    public double Value { get; set; }  
  }

Now, edit the “EventStream.cs” file by setting the properties set in the “OrderCreatedEventDTO”. We’ll modify the app service to use the “Add” method. This implementation indicates that we will be performing a “Subscribe” to the event called “OrderCreated”, which will be linked to the alias “Stream1” on the Stream adapter.
code src/Adapters/Stream/EventStream.cs

namespace DevPrime.Stream;
public class EventStream : EventStreamBase, IEventStream
{
    public override void StreamEvents()
    {
        
        Subscribe<IPaymentService, OrderCreatedEventDTO>("Stream1",
        "OrderCreated", (dto, paymentService, Dp) =>
        {
            Dp.Observability.Log("Evento recebido");

            var command = new Payment()
            {
                CustomerName = dto.CustomerName,
                OrderID = dto.OrderID,
                Value = dto.Value
            };
            paymentService.Add(command);           


        });
    }
}

The final step is to change the Stream adapter configuration file in the src/App/appsettings.json file and add the queue/topic name “orderevents” under “subscribe”, as per the example below:

  "DevPrime_Stream": [
    {      
      "Enable": "true",
      "Alias": "Stream1",
      "Default": "true",
      "StreamType": "Kafka",
      "HostName": "Localhost",
      "User": "",
      "Password": "",
      "Port": "9092",
      "Retry": "3",
      "Fallback": "State1",
      "Subscribe": [
        {
          "Topics": "orderevents"
        }
      ]
    }
  ],

Congratulations! You already have the “Payment” microservice configured to subscribe to the queue/topic named “OrderEvents”.

Asynchronous communication between microservices

Now you’ve reached the big time to run both microservices. Open each one in a window and run so that both have the services active. Go to the URL of the “Order” microservice in http://localhost:5000 and then go to “Post” and click on the “Try it out” button. Make a post and follow the incredible result, with the complete breakdown in each microservice and the “Payment” reacting to the event received.

Right now, we’re checking the processing result in the “Order” microservice, which receives a POST and emits the “OrderCreated” event. All details are automatically generated by the Devprime Observability adapter.

[INF][Web]["https://localhost:5001;http://localhost:5000"][Host]["Production"][Parameters]["Appsettings"][RequestId: 3282e6f8-d2bd-4ccf-934a-546276a83038]
[2021-06-19T20:23:07.654-03:00][INF][Web]["HTTP"][Order][Add]["{\n  \"customerName\": \"Ramon\",\n  \"customerTaxID\": \"string\",\n  \"itens\": [\n    {\n      \"description\": \"string\",\n      \"amount\": 0,\n      \"sku\": \"string\",\n      \"price\": 0\n    }\n  ],\n  \"total\": 0\n}"][Origin:"https://localhost:5001/swagger/index.html"][RequestId: 06721dfc-fd50-4783-8ea7-df201c22599b]
[INF][Application][OrderService][Add][RequestId: 06721dfc-fd50-4783-8ea7-df201c22599b]
[INF][Domain][Order][Add][RequestId: 06721dfc-fd50-4783-8ea7-df201c22599b]
[INF][Domain][Order][Handle][Event]["OrderCreated"][RequestId: 06721dfc-fd50-4783-8ea7-df201c22599b]
[INF][Application][EventHandler]["OrderCreatedEventHandler"][Event]["OrderCreated"][RequestId: 06721dfc-fd50-4783-8ea7-df201c22599b]
[INF][State][Type "MongoDB"][Alias "State1"][OrderRepository][Add][RequestId: 06721dfc-fd50-4783-8ea7-df201c22599b]
[INF][Stream][Type "Kafka"][Alias "Stream1"][Out][Event]["OrderCreated"]["Delivered"]["OrderEvents"]["{\"SagaId\":\"00000000-0000-0000-0000-000000000000\",\"Id\":\"35bc8937-deb8-4e35-a8c4-003171fd6e1b\",\"CorrelationId\":\"06721dfc-fd50-4783-8ea7-df201c22599b\",\"AppId\":\"3282e6f8-d2bd-4ccf-934a-546276a83038\",\"AppName\":\"ms-order\",\"Version\":1,\"Name\":\"OrderCreated\",\"When\":\"2021-06-19T20:23:07.8540357-03:00\",\"Payload\":{\"CustomerName\":\"Ramon\",\"CustomerTaxID\":\"string\",\"Total\":0,\"ID\":\"55a72f16-f4e5-476b-85d9-e86c6e6f390c\"}}"][RequestId: 06721dfc-fd50-4783-8ea7-df201c22599b]

And now the time comes for the ‘OrderCreated’ event in the “Payment” microservices.

[INF][Web]["https://localhost:5003;http://localhost:5002"][Host]["Production"][Parameters]["Appsettings"][RequestId: 072b12e7-c941-4b4b-9823-cbd6e4425309]
[INF][Stream][Type "Kafka"][Alias "Stream1"][In][Event]["OrderCreated"]["OrderEvents"]["{\"SagaId\":\"00000000-0000-0000-0000-000000000000\",\"Id\":\"35bc8937-deb8-4e35-a8c4-003171fd6e1b\",\"CorrelationId\":\"06721dfc-fd50-4783-8ea7-df201c22599b\",\"AppId\":\"3282e6f8-d2bd-4ccf-934a-546276a83038\",\"AppName\":\"ms-order\",\"Version\":1,\"Name\":\"OrderCreated\",\"When\":\"2021-06-19T20:23:07.8540357-03:00\",\"Payload\":{\"CustomerName\":\"Ramon\",\"CustomerTaxID\":\"string\",\"Total\":0,\"ID\":\"55a72f16-f4e5-476b-85d9-e86c6e6f390c\"}}"][RequestId: 35bc8937-deb8-4e35-a8c4-003171fd6e1b]
[INF][Application][PaymentService][Add][RequestId: 35bc8937-deb8-4e35-a8c4-003171fd6e1b]
[INF][Domain][Payment][Add][RequestId: 35bc8937-deb8-4e35-a8c4-003171fd6e1b]
[INF][Domain][Payment][Handle][Event]["PaymentCreated"][RequestId: 35bc8937-deb8-4e35-a8c4-003171fd6e1b]
[INF][Application][EventHandler]["PaymentCreatedEventHandler"][Event]["PaymentCreated"][RequestId: 35bc8937-deb8-4e35-a8c4-003171fd6e1b]
[INF][State][Type "MongoDB"][Alias "State1"][PaymentRepository][Add][RequestId: 35bc8937-deb8-4e35-a8c4-003171fd6e1b]

Congratulations! 🚀 You’ve just developed two microservices using Event-Driven with the Devprime platform and Kafka Stream.

To learn more: