Building an idempotent event-driven microservices

Creating a microservice to use in the example

We will use a microservice based on the Devprime platform

• Use an active account in Devprime and enable the Devprime CLI.
• Launch the containers from docker (MongoDB, RabbitMQ, and Redis)
• Create the orderevents and paymentevents queue in RabbitMQ.
• Create a new microservice following the example of “payment” presented.
• This microservice will listen to the queue/topic “orderevents”

Starting the idempotency setup

Setting idempotency on the “Devprime_App” switch
Change the Idempotency setting to “Enabled” to “true” and the “Flow” to “frontend”. We changed the “key” to “my-idempotency-key”. With this configuration, the API will require the idempotency key in Headar.
code .\src\app\appsettings.json

1
2
3
4
5
6
7
8
9

    "Idempotency": {
      "Enable": "true",
      "Alias": "State2",
      "Duration": "86400",
      "Flow": "backend",
      "key": "idempotency-key",
      "Scope": "all",
      "Action": "auto"
    }

Include a second persistence in the “Devprime_State” key
In this example, we’ll use a local Redis with a default password
code .\src\app\appsettings.json

1
2
3
4
5
6
7
8
9

{
  "enable": "true",
  "alias": "State2",
  "dbtype": "redis",
  "connection": "127.0.0.1:6379,password=LltF8Nx*yo",
  "timeout": "5",
  "retry": "2",
  "durationofbreak": "45"
}

For this scenario we will add the event directly into RabbitMQ to simulate a duplicate event.

Run the dp-stream microservice and sign in to the Rabbitmq portal at
http://localhost:15672 with the guest/guest user. Locate the Queues > orderevents menu and locate the “Publish message” item

Locate the payload field and paste the json below and then go to “Publish message” to queue that event.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

{
   "Headers":{
      "uber-trace-id":"126e838875d34dd3:5644a477603d0a55:bc20635b06c79bc2:1"
   },
   "Id":"36dbd9d0-7b8b-41fb-bdaf-2ff6a0672bd4",
   "CorrelationId":"18c71a48-0253-4247-9c8b-d576c4f8394e",
   "AppId":"21cef89c-229e-4291-8d6a-1ccf5c401484",
   "AppName":"ms-idempotency",
   "TenantID":"",
   "TenantUserID":"",
   "Version":1,
   "Name":"OrderCreated",
   "CreationDate":"2022-03-13T11:14:01.0284863-03:00",
   "Payload":{
      "ID":"ccfc8248-2ab5-4423-92fc-b16c4555add2",
      "CustomerName":"Bill",
      "CustomerTaxID":"string",
      "Total":100
   }
}

The microservice is listening to this queue and will react to the event and process it successfully.

Now go back to RabbitMQ and send the same event again and you will notice that when it reaches the microservice it is automatically rejected.

To repeat the test, change a value in the json payload by putting another guid in the ID, for example.

Starting Manual Idempotency

Devprime allows you to manually enter a specific method that will support idempotency. The first step is to change the main configuration and then go directly to the method to add the configuration.

Changing the “Action” key to “Manual”

1
2
3
4
5
6
7
8
9

    "Idempotency": {
      "Enable": "true",
      "Alias": "State2",
      "Duration": "86400",
      "Flow": "backend",
      "key": "idempotency-key",
      "Scope": "all",
      "Action": "manual"
    }

In manual mode, when performing a test by sending duplicate events, you will notice that they will be accepted. You need to visit each desired method and add “Dp.Idempotency.Enable = true;” to enable the idempotency control.

** Modifying the Stream for “Idempotency” **
Open the Event Hub and locate the subscribe of the desired event which in this context is the “OrderCreated” and add the “Dp.Idempotency.Enable = true;”.
code src\Adapters\Stream\EventStream.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

namespace Devprime.Stream;
public class EventStream : EventStreamBase, IEventStream
{
    public override void StreamEvents()
    {
        Subscribe<IPaymentService>("Stream1", "OrderCreated", (payload, paymentService, Dp) =>
        {
            Dp.Idempotency.Enable = true;

            var dto = System.Text.Json.JsonSerializer.Deserialize<OrderCreatedEventDTO>(payload);
            var command = new Payment()
            {
                CustomerName = dto.CustomerName,
                OrderID = dto.OrderID,
                Value = dto.Value
            };
            paymentService.Add(command);
        });
    }
}

When you try to run duplicate events again, you’ll notice that it’s working again.

More articles on idempotency

Idempotency in microservices and REST APIs
Building an idempotent event-driven microservices