Devprime | Documentation – Stream

Settings: Communication and integration protocol

Mon, 01 Jan 0001 00:00:00 +0000

This documentation defines the payload model for publishing events to streaming systems such as RabbitMQ and Kafka, and details the adaptations required for legacy applications, highlighting the required and optional fields. For external applications, it is essential that the payload follows the same format and includes the specified information, ensuring compatibility and efficient monitoring. While native integration is optional, it is recommended to maximize the benefits of Devprime across the application ecosystem.

Event Default Payload Structure

{
  "Headers": {},
  "Id": "<Guid>", 
  "CorrelationId": "<Guid>",
  "TraceId": "<Guid>",
  "AppId": "<Guid fixo identificando app legado>",
  "AppName": "Myapp",
  "Version": 1,
  "Name": "OrderCreated",
  "CreationDate": "2022-06-07T11:26:21.7636769-03:00", 
  "Payload": {
    //Include your custom payload (json)
  }
}

Description of Fields

Field	Type	Description	Obligation
Headers	`object`	Contains additional headers, can be used to carry specific metadata.	Optional
Id	`string (GUID)`	Unique Guid identifier of the event generated by the legacy application when creating a new event.	Required
CorrelationId	`string (GUID)`	Guid identifier used to correlate events and must be generated by the legacy application.	Required
TraceId	`string (GUID)`	Tracking Guid identifier, useful for distributed tracing.	Required
AppId	`string (GUID)`	Fixed Guid identifier of the legacy application that generates the event.	Required
AppName	`string`	Name of the application that is publishing the event. Example: “Integrator 1.0”.	Required
Version	`integer`	Event version. Usually incremented in case of changes in the payload format.	Required
Name	`string`	Event name. Example: “OrderCreated”.	Required
CreateDate	`string (DateTime)`	Date and time of creation of the event, in ISO 8601 format.	Required
Payload	`object (JSON)`	JSON file containing the event data.	Required

Examples of Use

Full Payload (Recommended)

This example includes all the fields required for complete traceability and proper integration with the observability system:

{
  "Headers": {},
  "Id": "6598cd76-7e9f-4f1f-a140-01ed92f3b60d",
  "CorrelationId": "26a91e94-7f31-4401-8352-3d6d7ad0f5fb",
  "TraceId": "26a91e94-7f31-4401-8352-3d6d7ad0f5fb",
  "AppId": "e6ae2fd2-99ca-4287-bcaa-fb8c83b92edc",
  "AppName": "MyApp",
  "TenantID": "",
  "TenantUserID": "",
  "Version": 1,
  "Name": "OrderCreated",
  "CreationDate": "2024-09-11T12:04:47.5302497-03:00",
  "Payload": {
    "OrderId": "9375cde2-1770-4186-8ffb-ee6e81aa5d91",
    "CustomerName": "Name",
    "CustomerTaxId": "string",
    "Total": 0
  }
}

Reduced Payload (Optional)

This example omits some fields for a simpler implementation for cases where traceability is not required:

{
  "AppId": "fef824a5-0eea-4b49-b08e-0bad25dbd55d",
  "AppName": "MyApp",
  "Version": 1,
  "Name": "OrderCreated",
  "CreationDate": "2022-06-07T11:26:21.7636769-03:00",
  "Payload": {
    "OrderId": "9375cde2-1770-4186-8ffb-ee6e81aa5d91",
    "CustomerName": "Name",
    "CustomerTaxId": "string",
    "Total": 0
  }
}

Final Thoughts

Implementing the payload pattern described above will allow legacy and external applications to interact effectively with the Devprime platform, ensuring that events are properly monitored and tracked throughout the homologation and production process. It is strongly recommended that developers follow the guidance to ensure compatibility and integration efficiency.

To learn more:

Settings: Enabling events on the Steam Adapter

Mon, 01 Jan 0001 00:00:00 +0000

Introduction to Event Subscription in the Stream Adapter

The first step in enabling event subscription for a queue or topic is to configure the Subscribe settings in the Stream Adapter Configuration. When you start the microservice, it will automatically connect to the defined active queues. However, if an event arrives and there is no Subscribe setting, the event is rejected by the Stream Adapter.

Queued subscription notifications using the RabbitMQ protocol

"DevPrime_Stream": [
    {
      "Alias": "Stream1",
      "Enable": "true",
      "Default": "true",
      "StreamType": "RabbitMQ",
      "HostName": "Localhost",
      "User": "guest",
      "Password": "guest",
      "Port": "5672",
      "Exchange": "devprime",
      "ExchangeType": "direct",
      "Retry": "3",
      "Fallback": "State1",
      "Threads": "30",
      "Buffer": "1",
      "Subscribe": [
        {
          "Queues": "ordernotifications"
        }
      ]
    }
  ],

If you try to run the microservice and the queue or topic doesn’t exist, you’ll encounter an initial error in the log.

[ERR][Stream][Type "RabbitMQ"][Alias "Stream1"][Subscribe]["ordernotifications not found"][RID c57842d3-555e-4755-9b95-1457c9127160][TID c57842d3-555e-4755-9b95-1457c9127160]

When an unimplemented event is received by the Stream Adapter, it will be automatically rejected and the initial error will be logged.

[INF][Stream][Type "RabbitMQ"][Alias "Stream1"][In][Event]["OrderUpdated"]["Rejected"]["Event subscribe missing"]["{\"AppId\":\"fef824a5-0eea-4b49-b08e-0bad25dbd55d\",\"AppName\":\"MyApp\",\"Version\":1,\"Name\":\"OrderUpdated\",\"CreationDate\":\"2022-06-07T11:26:21.7636769-03:00\",\"Payload\":{\"OrderId\":\"9375cde2-1770-4186-8ffb-ee6e81aa5d91\",\"CustomerName\":\"Name\",\"CustomerTaxId\":\"string\",\"Total\":0}}"][RID e3d1f4b2-cf41-494d-a3df-58096e5bb0c9][TID e3d1f4b2-cf41-494d-a3df-58096e5bb0c9]

Enabling event reception on the Stream Adapter

The Stream configuration is done in the EventStream.cs file, located in the Stream Adapter project, and can be opened directly in Visual Studio Code to implement the routing rules for incoming events. The Adapter is compatible with all technologies supported by Devprime and allows you to receive events from multiple streams simultaneously.

Navigate to the project folder and open it in Visual Studio Code to find where to apply the settings.

`1`	`code src/adapters/Stream/EventStream.cs`

using Application.Interfaces.Services;
using Application.Services.Order.Model;

namespace DevPrime.Stream;

public class EventStream : EventStreamBase, IEventStream
{
    // Override the method to handle streaming events
    public override void StreamEvents()
    {
        // Mapping and receiving the "OrderUpdated" event from Stream1
        Subscribe<IOrderService, OrderCreatedEventDTO>("Stream1",
         "OrderUpdated", (dto, orderService, Dp) =>
        {
            // Log received data from the "OrderUpdated" event
            Dp.Observability.Log($"[Stream] Received data: Customer:
            {dto.CustomerName}; TaxId:{dto.CustomerTaxID};
            Total:{dto.Total}");    
        });
    }
}

In the log below, you can track the details of the data received.

[INF][Stream][Type "RabbitMQ"][Alias "Stream1"][In][Event]["OrderUpdated"]["ordernotifications"]["{\"Id\":\"43669773-229c-45b2-8e5c-99a248e0fe34\",\"CorrelationId\":\"43669773-229c-45b2-8e5c-99a248e0fe34\",\"TraceId\":\"43669773-229c-45b2-8e5c-99a248e0fe34\",\"AppId\":\"48d06d35-c7b1-4ed7-b393-1b249d32eb9b\",\"AppName\":\"order\",\"Version\":1,\"Name\":\"OrderUpdated\",\"CreationDate\":\"2022-06-07T11:26:21.7636769-03:00\",\"Payload\":{\"OrderId\":\"9375cde2-1770-4186-8ffb-ee6e81aa5d91\",\"CustomerName\":\"Joe Demo\",\"CustomerTaxID\":\"XYZ9909\",\"Total\":12899}}"][RID 43669773-229c-45b2-8e5c-99a248e0fe34][TID 43669773-229c-45b2-8e5c-99a248e0fe34]
[INF][Custom][[Stream] Received data: Customer:Joe Demo; TaxId:XYZ9909; Total:12899][RID 43669773-229c-45b2-8e5c-99a248e0fe34][TID 43669773-229c-45b2-8e5c-99a248e0fe34]

Receiving different events in the Stream Adapter

In this scenario, we will be dealing with “OrderUpdated” and “OrderCanceled”. Other even valid events will be dropped by the event hub on the Stream Adapter.

Navigate to the project folder and open it in Visual Studio Code to find where to apply the settings.

`1`	`code src/adapters/Stream/EventStream.cs`

public override void StreamEvents()
{
    // Mapping and receiving the "OrderUpdated" event from Stream1
    Subscribe<IOrderService, OrderCreatedEventDTO>("Stream1",
     "OrderUpdated", (dto, orderService, Dp) =>
    {
        // Log received data from the "OrderUpdated" event
        Dp.Observability.Log($"[Stream] Received data: Customer:
        {dto.CustomerName}; TaxId:{dto.CustomerTaxID};
         Total:{dto.Total}");    
    });

    // Mapping and receiving the "OrderCanceled" event from Stream1
    Subscribe<IOrderService, OrderCreatedEventDTO>("Stream1",
     "OrderCanceled", (dto, orderService, Dp) =>
    {
        // Log received data from the "OrderCanceled" event
        Dp.Observability.Log($"[Stream] Received data: Customer:
        {dto.CustomerName}; TaxId:{dto.CustomerTaxID}; Total:{dto.Total}");    
    });

}

Receiving events from legacy applications in the Stream Adapter

Devprime provides a standard protocol for asynchronous communication between microservices using event-driven architecture. However, to integrate with legacy applications, you may need to handle generic events. To do this, enable the functionality in the Stream Adapter event hub to intercept non-standard payloads.

Example payload for queued submission and use as a generic example:

{
  "IntegrationId": "9b3a4c56-d0d4-4a1a-b0b7-3c1e4d5a6f77",
  "CustomerId": "1a2b3c4d-5e6f-7g8h-9i0j-k1l2m3n4o5p6",
  "ComplianceApprovedId": "e7f8g9h0-i1j2-k3l4-m5n6-o7p8q9r0s1t2",
  "Balance": 12345.67
}

The default procedure will be auto-rejection, as the event will not be identified as a standard Devprime event. This will be the default flow:

[INF][Stream][Type "RabbitMQ"][Alias "Stream1"][In][Event]["Generic"]["Rejected"]["Subscribe missing"]["{\"IntegrationId\":\"9b3a4c56-d0d4-4a1a-b0b7-3c1e4d5a6f77\",\"CustomerId\":\"1a2b3c4d-5e6f-7g8h-9i0j-k1l2m3n4o5p6\",\"ComplianceApprovedId\":\"e7f8g9h0-i1j2-k3l4-m5n6-o7p8q9r0s1t2\",\"Balance\":12345.67} RabbitMQ event receive error [Exception of type 'System.Text.Json.JsonException' was thrown.]"]

Enabling Receiving Legacy Events on the Stream Adapter
Open the project in Visual Studio Code, apply the necessary settings, and add a generic processing flow using “Subscribe<IOrderService>((dto, orderService, Dp))” to accept events from legacy systems with protocols other than the Devprime standard. In this model, do not use a DTO OrderCreatedEventDTO for automatic conversion or identify the incoming event with filters such as “OrderUpdated” or “OrderCanceled”.

`1`	`code src/adapters/Stream/EventStream.cs`

public override void StreamEvents()
{
    // Mapping and receiving the "OrderUpdated" event from Stream1
    Subscribe<IOrderService, OrderCreatedEventDTO>("Stream1",
     "OrderUpdated", (dto, orderService, Dp) =>
    {
        // Log received data from the "OrderUpdated" event
        Dp.Observability.Log($"[Stream] Received data: 
        Customer:{dto.CustomerName}; 
        TaxId:{dto.CustomerTaxID};
        Total:{dto.Total}");    
    });

    // Mapping and receiving the "OrderCanceled" event from Stream1
    Subscribe<IOrderService, OrderCreatedEventDTO>("Stream1",
     "OrderCanceled", (dto, orderService, Dp) =>
    {
        // Log received data from the "OrderCanceled" event
        Dp.Observability.Log($"[Stream] Received data:
         Customer:{dto.CustomerName}; 
         TaxId:{dto.CustomerTaxID}; 
         Total:{dto.Total}");    
    });

    // Receiving generic event from Legacy Systems
    Subscribe<IOrderService>((dto, orderService, Dp) =>
    { 
      // Log received data from the Generic event
        Dp.Observability.Log(dto); 
    });  

}

When you run the microservice with the new configuration that supports legacy application events, you can track the successful receipt in the log below, followed by the processing breakdown. In this model, you must follow the procedure of structuring the manual conversion of the received data, passing it through the model to the Application Services layer, which will direct the decision in the business rule in the Domain layer.

[INF][Stream][Type "RabbitMQ"][Alias "Stream1"][In][Event]["Generic"]["ordernotifications"]["{\"Id\":\"a67a03b3-19ad-4239-a394-2e8a152ac125\",\"CorrelationId\":\"a67a03b3-19ad-4239-a394-2e8a152ac125\",\"TraceId\":\"3a9a0185-c345-4aa7-a2c5-ee54880b9d72\",\"AppId\":\"f6e8b14e-94d2-4b6c-9665-1657ae628237\",\"AppName\":\"order\",\"Name\":\"Generic\",\"Payload\":\"{\"IntegrationId\":\"9b3a4c56-d0d4-4a1a-b0b7-3c1e4d5a6f77\",\"CustomerId\":\"1a2b3c4d-5e6f-7g8h-9i0j-k1l2m3n4o5p6\",\"ComplianceApprovedId\":\"e7f8g9h0-i1j2-k3l4-m5n6-o7p8q9r0s1t2\",\"Balance\":12345.67}\"}"]

[INF][Custom][{"IntegrationId":"9b3a4c56-d0d4-4a1a-b0b7-3c1e4d5a6f77","CustomerId":"1a2b3c4d-5e6f-7g8h-9i0j-k1l2m3n4o5p6","ComplianceApprovedId":"e7f8g9h0-i1j2-k3l4-m5n6-o7p8q9r0s1t2","Balance":12345.67}]

To learn more:

Settings: Enabling Multistream Communication

Mon, 01 Jan 0001 00:00:00 +0000

By adopting the Multistream strategy through the configuration of the Stream Adapter, it is possible to activate, for example, RabbitMQ and Kafka at the same time to meet specific scenarios. All events received by the Stream Adapter are processed in the StreamEvents() method offered by the Stream Adapter on the Devprime platform.

 "Devprime_Stream": [
    {
      "Alias": "Stream1",
      "Enable": "true",
      "Default": "fakse",
      "StreamType": "RabbitMQ",
      "HostName": "Localhost",
      "User": "guest",
      "Password": "guest",
      "Port": "5672",
      "Exchange": "devprime",
      "ExchangeType": "direct",
      "Retry": "3",
      "Fallback": "State1",
      "Threads": "30",
      "Buffer": "5",
      "Subscribe": []
    },
    {
      "Alias": "Stream2",
      "Enable": "true",
      "Default": "true",
      "StreamType": "Kafka",
      "Hostname": "Localhost",
      "Port": "9092",
      "User": "",
      "Password": "",
      "Retry": "3",
      "Fallback": "State1",
      "Subscribe":[]
    }
  ],

Example in the application log with this configuration:

1
2

[INF][Stream][Type "RabbitMQ"][Alias "Stream1"]["Enable"]
[INF][Stream][Type "Kafka"][Alias "Stream2"]["Enable"]

YAML example with application settings:
To export the configurations as an environment variable and use in Kubernetes, run the command in the Devprime CLI:
dp export kubernetes

- name: devprime_stream1
  value: "alias=Stream1|||enable=true|||default=true|||streamtype=RabbitMQ|||
  hostname=rabbitmq.default.svc|||user=guest|||password=guest|||port=5672|||
  exchange=devprime|||exchangetype=direct|||retry=3|||fallback=State1|||
  threads=30|||buffer=1" 
- name: devprime_stream2
  value: "alias=Stream2|||enable=true|||default=false|||streamtype=Kafka|||
  hostname=kafka.default.svc|||port=9092|||retry=3|||fallback=State1"

To learn more:

Settings: RabbitMQ

Mon, 01 Jan 0001 00:00:00 +0000

Settings: Kafka

Mon, 01 Jan 0001 00:00:00 +0000

Apache Kafka can be used on-premise, cloud, docker, kubernetes, and managed platforms such as Confluent Cloud, Azure Event Hub, and others.

Setting up the Stream with Kafka

The configuration example below demonstrates the use of Apache Kafka defined with Alias “Stream1” and as the default for sending events. By enabling the “Subscribe” item, we are subscribing to this topic and receiving events in the Stream Adapter Hub.

General	—
Enable	Enable the Stream adapter (True/False)
Alias	Define a unique name to identify the service
Default	Sets the default for the centary to have more than one Stream (True/False)
StreamType	Sets the Stream type (RabbitMQ
HostName	Configure the Kafka
User	Configures the user
Password	Configures the user’s password
Port	Configure the port
Retry	Sets the number of retries
Fallback	Sets a State for automatic persistence in case of failure
Subscribe	Configures Kafka

Standard Kafka Configuration Example

  "DevPrime_Stream": [
    {      
      "Enable": "true",
      "Alias": "Stream1",
      "Default": "true",
      "StreamType": "Kafka",
      "HostName": "Localhost",
      "User": "",
      "Password": "",
      "Port": "9092",
      "Retry": "3",
      "Fallback": "State1",
      "Subscribe": []
    }
  ],

Example of Kafka configuration with subscription in the topics: notifications and paymentevents
When adding more than one topic, it is necessary to separate them by a comma “,” as demonstrated.

  "DevPrime_Stream": [
    {      
      "Enable": "true",
      "Alias": "Stream1",
      "Default": "true",
      "StreamType": "Kafka",
      "HostName": "Localhost",
      "User": "",
      "Password": "",
      "Port": "9092",
      "Retry": "3",
      "Fallback": "State1",
      "Subscribe": [
        {
          "Topics": "notifications,paymentevents"
        }
      ]
    }
  ],

Example in the application log with this configuration:

`1`	`[INF][Stream][Type "Kafka"][Alias "Stream1"]["Enable"][Subscribe]["orderevents"]`

YAML example with application settings:

- name: devprime_stream1
    value: "alias=Stream1|||enable=true|||default=true|||streamtype=Kafka|||
    hostname=kafka.default.svc|||port=9092|||retry=3|||fallback=State1|||
    subscribe=[topics=orderevents]"

Settings: CloudAMQP

Mon, 01 Jan 0001 00:00:00 +0000

CloudAMQP’s configurations follow the same model as RabbitMQ’s with a point of attention to the HostName parameter that already includes the url and credentials provided when creating the service in CloudAMQP. User/Password/Port parameters are not required

The JSON below demonstrates the default configuration with the RabbitMQ credentials, the exchange used, the Retry, the Fallback, the Threads and the ‘Subscribe’ option to receive events in the ‘orderevents’ queue as defined in the Queues item.

"DevPrime_Stream": [
    {
      "Enable": "true",
      "Alias": "Stream1",
      "Default": "true",
      "StreamType": "RabbitMQ",
      "HostName": "amqps://user:password@demo.rmq.cloudamqp.com/demo",
      "Exchange": "devprime",
      "ExchangeType": "direct",
      "Retry": "3",
      "Fallback": "State1",
      "Threads": "30",
      "Buffer": "1",
      "Subscribe": [
        {
          "Queues": "orderevents"
        }
      ]
    }
  ],

In the example below, we’re detailing the default Adapter settings and the RabbitMQ-specific settings.

General	—
Enable	Enable the Stream adapter (True/False)
Alias	Define a unique name to identify the service
Default	Sets the default for the centary to have more than one Stream (True/False)
StreamType	Sets the Stream Type (RabbitMQ/Kafka)
HostName	Configures the url/credential provided by CloudAMQP
Exchange	Configures the default RabbitMQ
ExchangeType	Configures the Exchange type
Retry	Sets the number of retries
Fallback	Sets a State for automatic persistence in case of failure
Threads	Sets the number of concurrent connections processing events
Buffer	Sets the number of events obtained per connection
Subscribe	Configures RabbitMQ queues for read

Example in the application log with this configuration:

1
2

[INF][ms-order][Stream][Type "RabbitMQ"][Alias "Stream1"]
["Enable"][Subscribe]["orderevents"][Parallel "30"][Buffer "1"]

YAML example with application settings:
To export the configurations as an environment variable and use in Kubernetes, run the command in the Devprime CLI:
dp export kubernetes

- name: devprime_stream1
    value: "alias=Stream1|||enable=true|||default=true|||streamtype=RabbitMQ|||
    hostname=amqps://user:password@demo.rmq.cloudamqp.com/demo|||
    exchange=devprime|||exchangetype=direct|||retry=3|||
    fallback=State1|||threads=30|||buffer=1|||
    subscribe=[queues=orderevents]"

Use the RabbitMQ examples to explore other variations in the configuration.

Settings: Azure Service Bus

Mon, 01 Jan 0001 00:00:00 +0000

Settings: Google Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

  "DevPrime_Stream": [
    {
      "Alias": "Stream1",
      "Enable": "true",
      "Default": "true",
      "StreamType": "GooglePubSub",
      "Project_ID": "",
      "Private_Key_ID": "",
      "Private_Key": "",
      "Client_Email": "",
      "Client_ID": "",
      "Auth_URI": "",
      "Token_URI": "",
      "Auth_Provider_x509_Cert_Url": "",
      "Client_X509_Cert_Url": "",
      "Retry": "3",
      "Fallback": "State1",
      "Threads": "10",
      "Subscribe": []
    }
  ],

Settings: AWS SQS

Mon, 01 Jan 0001 00:00:00 +0000

Devprime | Documentation – Stream

Settings: Communication and integration protocol

Event Default Payload Structure

Description of Fields

Examples of Use

Full Payload (Recommended)

Reduced Payload (Optional)

Final Thoughts

Settings: Enabling events on the Steam Adapter

Introduction to Event Subscription in the Stream Adapter

Enabling event reception on the Stream Adapter

Receiving different events in the Stream Adapter

Receiving events from legacy applications in the Stream Adapter

Settings: Enabling Multistream Communication

Settings: RabbitMQ

Settings: Kafka

Setting up the Stream with Kafka

Settings: CloudAMQP

Subscribe to a queue in RabbitMQ

Settings: Azure Service Bus

Settings: Google Pub/Sub

Settings: AWS SQS